[==========] Running 5 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240426 09:53:37.307547 8758 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.8.141.190:37867
I20240426 09:53:37.308701 8758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 09:53:37.309638 8758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:53:37.325523 8764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:53:37.325811 8767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:53:37.326342 8765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:53:37.327096 8758 server_base.cc:1034] running on GCE node
I20240426 09:53:38.362234 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 09:53:38.362454 8758 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 09:53:38.362609 8758 hybrid_clock.cc:648] HybridClock initialized: now 1714125218362594 us; error 0 us; skew 500 ppm
I20240426 09:53:38.363401 8758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:53:38.368368 8758 webserver.cc:460] Webserver started at http://127.8.141.190:34049/ using document root <none> and password file <none>
I20240426 09:53:38.369163 8758 fs_manager.cc:359] Metadata directory not provided
I20240426 09:53:38.369352 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:53:38.369709 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:53:38.375576 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/master-0-root/instance:
uuid: "af87027f7d3c4ebfb3d5a5daa8cfc402"
format_stamp: "Formatted at 2024-04-26 09:53:38 on dist-test-slave-5txv"
I20240426 09:53:38.382995 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20240426 09:53:38.387637 8773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 09:53:38.388659 8758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240426 09:53:38.388962 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/master-0-root
uuid: "af87027f7d3c4ebfb3d5a5daa8cfc402"
format_stamp: "Formatted at 2024-04-26 09:53:38 on dist-test-slave-5txv"
I20240426 09:53:38.389266 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:53:38.441042 8758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 09:53:38.442376 8758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 09:53:38.442813 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:53:38.515604 8758 rpc_server.cc:307] RPC server started. Bound to: 127.8.141.190:37867
I20240426 09:53:38.515684 8824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.141.190:37867 every 8 connection(s)
I20240426 09:53:38.523352 8825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:38.547211 8825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402: Bootstrap starting.
I20240426 09:53:38.552949 8825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:38.554675 8825 log.cc:824] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402: Log is configured to *not* fsync() on all Append() calls
I20240426 09:53:38.559262 8825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402: No bootstrap required, opened a new log
I20240426 09:53:38.582468 8825 raft_consensus.cc:348] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af87027f7d3c4ebfb3d5a5daa8cfc402" member_type: VOTER }
I20240426 09:53:38.583065 8825 raft_consensus.cc:374] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:38.583323 8825 raft_consensus.cc:729] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af87027f7d3c4ebfb3d5a5daa8cfc402, State: Initialized, Role: FOLLOWER
I20240426 09:53:38.584081 8825 consensus_queue.cc:260] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af87027f7d3c4ebfb3d5a5daa8cfc402" member_type: VOTER }
I20240426 09:53:38.584579 8825 raft_consensus.cc:388] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240426 09:53:38.584851 8825 raft_consensus.cc:482] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240426 09:53:38.585672 8825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:38.589485 8825 raft_consensus.cc:504] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af87027f7d3c4ebfb3d5a5daa8cfc402" member_type: VOTER }
I20240426 09:53:38.590160 8825 leader_election.cc:302] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af87027f7d3c4ebfb3d5a5daa8cfc402; no voters:
I20240426 09:53:38.592221 8825 leader_election.cc:288] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240426 09:53:38.592525 8828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:53:38.594933 8828 raft_consensus.cc:686] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [term 1 LEADER]: Becoming Leader. State: Replica: af87027f7d3c4ebfb3d5a5daa8cfc402, State: Running, Role: LEADER
I20240426 09:53:38.595734 8828 consensus_queue.cc:237] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af87027f7d3c4ebfb3d5a5daa8cfc402" member_type: VOTER }
I20240426 09:53:38.597004 8825 sys_catalog.cc:564] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [sys.catalog]: configured and running, proceeding with master startup.
I20240426 09:53:38.607969 8830 sys_catalog.cc:455] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af87027f7d3c4ebfb3d5a5daa8cfc402. Latest consensus state: current_term: 1 leader_uuid: "af87027f7d3c4ebfb3d5a5daa8cfc402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af87027f7d3c4ebfb3d5a5daa8cfc402" member_type: VOTER } }
I20240426 09:53:38.608119 8829 sys_catalog.cc:455] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af87027f7d3c4ebfb3d5a5daa8cfc402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af87027f7d3c4ebfb3d5a5daa8cfc402" member_type: VOTER } }
I20240426 09:53:38.608611 8830 sys_catalog.cc:458] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [sys.catalog]: This master's current role is: LEADER
I20240426 09:53:38.608700 8829 sys_catalog.cc:458] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [sys.catalog]: This master's current role is: LEADER
I20240426 09:53:38.612147 8838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240426 09:53:38.624886 8838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240426 09:53:38.626971 8758 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240426 09:53:38.640944 8838 catalog_manager.cc:1347] Generated new cluster ID: 12ebbb3dfd4f4f3db8c649d337b08bd5
I20240426 09:53:38.641237 8838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240426 09:53:38.661132 8838 catalog_manager.cc:1370] Generated new certificate authority record
I20240426 09:53:38.663004 8838 catalog_manager.cc:1504] Loading token signing keys...
I20240426 09:53:38.676646 8838 catalog_manager.cc:5857] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402: Generated new TSK 0
I20240426 09:53:38.677424 8838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240426 09:53:38.693131 8758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:53:38.701949 8847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:53:38.705227 8848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:53:38.707165 8850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:53:38.708012 8758 server_base.cc:1034] running on GCE node
I20240426 09:53:38.708832 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 09:53:38.709085 8758 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 09:53:38.709295 8758 hybrid_clock.cc:648] HybridClock initialized: now 1714125218709276 us; error 0 us; skew 500 ppm
I20240426 09:53:38.710119 8758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:53:38.714692 8758 webserver.cc:460] Webserver started at http://127.8.141.129:42219/ using document root <none> and password file <none>
I20240426 09:53:38.715241 8758 fs_manager.cc:359] Metadata directory not provided
I20240426 09:53:38.715428 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:53:38.715687 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:53:38.716768 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/ts-0-root/instance:
uuid: "cdbe92f5959a47dda817bbec2148eab9"
format_stamp: "Formatted at 2024-04-26 09:53:38 on dist-test-slave-5txv"
I20240426 09:53:38.722746 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240426 09:53:38.728041 8855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 09:53:38.728780 8758 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20240426 09:53:38.729035 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/ts-0-root
uuid: "cdbe92f5959a47dda817bbec2148eab9"
format_stamp: "Formatted at 2024-04-26 09:53:38 on dist-test-slave-5txv"
I20240426 09:53:38.729296 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:53:38.748766 8758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 09:53:38.749890 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:53:38.752781 8758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:53:38.757074 8758 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:53:38.757376 8758 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 09:53:38.757683 8758 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:53:38.757902 8758 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 09:53:38.854815 8758 rpc_server.cc:307] RPC server started. Bound to: 127.8.141.129:42507
I20240426 09:53:38.854898 8917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.141.129:42507 every 8 connection(s)
I20240426 09:53:38.861428 8758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:53:38.871398 8922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:53:38.875357 8923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:53:38.878798 8925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:53:38.880765 8758 server_base.cc:1034] running on GCE node
I20240426 09:53:38.881785 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 09:53:38.882040 8758 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 09:53:38.882233 8758 hybrid_clock.cc:648] HybridClock initialized: now 1714125218882218 us; error 0 us; skew 500 ppm
I20240426 09:53:38.883033 8758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:53:38.884066 8918 heartbeater.cc:347] Connected to a master server at 127.8.141.190:37867
I20240426 09:53:38.884516 8918 heartbeater.cc:464] Registering TS with master...
I20240426 09:53:38.885872 8918 heartbeater.cc:511] Master 127.8.141.190:37867 requested a full tablet report, sending...
I20240426 09:53:38.886788 8758 webserver.cc:460] Webserver started at http://127.8.141.130:41037/ using document root <none> and password file <none>
I20240426 09:53:38.887442 8758 fs_manager.cc:359] Metadata directory not provided
I20240426 09:53:38.887691 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:53:38.888008 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:53:38.888639 8790 ts_manager.cc:194] Registered new tserver with Master: cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507)
I20240426 09:53:38.889400 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/ts-1-root/instance:
uuid: "9e98a77341c74d6d90d01cd3e39586b3"
format_stamp: "Formatted at 2024-04-26 09:53:38 on dist-test-slave-5txv"
I20240426 09:53:38.890538 8790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52444
I20240426 09:53:38.895227 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240426 09:53:38.900063 8930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 09:53:38.900807 8758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20240426 09:53:38.901043 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/ts-1-root
uuid: "9e98a77341c74d6d90d01cd3e39586b3"
format_stamp: "Formatted at 2024-04-26 09:53:38 on dist-test-slave-5txv"
I20240426 09:53:38.901297 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:53:38.935716 8758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 09:53:38.937292 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:53:38.939353 8758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:53:38.942402 8758 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:53:38.942574 8758 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 09:53:38.942831 8758 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:53:38.942977 8758 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 09:53:39.040287 8758 rpc_server.cc:307] RPC server started. Bound to: 127.8.141.130:35991
I20240426 09:53:39.040493 8992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.141.130:35991 every 8 connection(s)
I20240426 09:53:39.049312 8758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:53:39.058274 8996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:53:39.061303 8997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:53:39.066469 8993 heartbeater.cc:347] Connected to a master server at 127.8.141.190:37867
I20240426 09:53:39.066886 8993 heartbeater.cc:464] Registering TS with master...
I20240426 09:53:39.067862 8993 heartbeater.cc:511] Master 127.8.141.190:37867 requested a full tablet report, sending...
I20240426 09:53:39.070152 8758 server_base.cc:1034] running on GCE node
I20240426 09:53:39.070600 8790 ts_manager.cc:194] Registered new tserver with Master: 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991)
W20240426 09:53:39.071257 8999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:53:39.072078 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 09:53:39.072297 8758 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 09:53:39.072315 8790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52460
I20240426 09:53:39.072531 8758 hybrid_clock.cc:648] HybridClock initialized: now 1714125219072511 us; error 0 us; skew 500 ppm
I20240426 09:53:39.073137 8758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:53:39.075904 8758 webserver.cc:460] Webserver started at http://127.8.141.131:39887/ using document root <none> and password file <none>
I20240426 09:53:39.076349 8758 fs_manager.cc:359] Metadata directory not provided
I20240426 09:53:39.076534 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:53:39.076764 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:53:39.077809 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/ts-2-root/instance:
uuid: "d99df4d00f1e42699f4e40da65bbc33d"
format_stamp: "Formatted at 2024-04-26 09:53:39 on dist-test-slave-5txv"
I20240426 09:53:39.082124 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240426 09:53:39.085212 9004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 09:53:39.085991 8758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240426 09:53:39.086241 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/ts-2-root
uuid: "d99df4d00f1e42699f4e40da65bbc33d"
format_stamp: "Formatted at 2024-04-26 09:53:39 on dist-test-slave-5txv"
I20240426 09:53:39.086498 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714125217154154-8758-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:53:39.100610 8758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 09:53:39.101732 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:53:39.104075 8758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:53:39.106598 8758 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:53:39.106870 8758 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240426 09:53:39.107162 8758 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:53:39.107307 8758 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 09:53:39.196722 8758 rpc_server.cc:307] RPC server started. Bound to: 127.8.141.131:43019
I20240426 09:53:39.196813 9066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.141.131:43019 every 8 connection(s)
I20240426 09:53:39.213455 9067 heartbeater.cc:347] Connected to a master server at 127.8.141.190:37867
I20240426 09:53:39.213763 9067 heartbeater.cc:464] Registering TS with master...
I20240426 09:53:39.214432 9067 heartbeater.cc:511] Master 127.8.141.190:37867 requested a full tablet report, sending...
I20240426 09:53:39.216854 8790 ts_manager.cc:194] Registered new tserver with Master: d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:39.217687 8758 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01613481s
I20240426 09:53:39.218216 8790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52464
I20240426 09:53:39.221530 8758 test_util.cc:268] Using random seed: -523579389
I20240426 09:53:39.268985 8790 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52480:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240426 09:53:39.272349 8790 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240426 09:53:39.379348 8958 tablet_service.cc:1467] Processing CreateTablet for tablet 38c779579a0648ee982d552a8fec0bec (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240426 09:53:39.381078 8958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c779579a0648ee982d552a8fec0bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.393985 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 91ee2df31bb64b6a9536f91149a87c0d (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240426 09:53:39.394022 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfc67501c0a441d9ac98720108b6f73 (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240426 09:53:39.395845 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ee2df31bb64b6a9536f91149a87c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.413852 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 69d609ac70eb4a09b5bc56973c5d3009 (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240426 09:53:39.412541 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfc67501c0a441d9ac98720108b6f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.420550 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d609ac70eb4a09b5bc56973c5d3009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.424840 8954 tablet_service.cc:1467] Processing CreateTablet for tablet 4af8a55ea6e5429093a5ec516bd3f6a0 (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240426 09:53:39.428226 8954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af8a55ea6e5429093a5ec516bd3f6a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.431035 8952 tablet_service.cc:1467] Processing CreateTablet for tablet df73379416114f8ab7efd3caf9c22e7e (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240426 09:53:39.434734 8952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df73379416114f8ab7efd3caf9c22e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.434547 8953 tablet_service.cc:1467] Processing CreateTablet for tablet 431d4df7c67747fcb0812c91964c801b (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240426 09:53:39.433844 8951 tablet_service.cc:1467] Processing CreateTablet for tablet 97deaa8654f64ad2a63314ffaa91d955 (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240426 09:53:39.437362 8951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97deaa8654f64ad2a63314ffaa91d955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.438943 8953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431d4df7c67747fcb0812c91964c801b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.442114 8950 tablet_service.cc:1467] Processing CreateTablet for tablet 7b463d0d799d4c77abcbb023cc934bf2 (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240426 09:53:39.445304 8950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b463d0d799d4c77abcbb023cc934bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.463065 9085 tablet_bootstrap.cc:492] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3: Bootstrap starting.
I20240426 09:53:39.473961 9085 tablet_bootstrap.cc:654] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:39.495620 8882 tablet_service.cc:1467] Processing CreateTablet for tablet 91ee2df31bb64b6a9536f91149a87c0d (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240426 09:53:39.496670 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 69d609ac70eb4a09b5bc56973c5d3009 (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240426 09:53:39.502692 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d609ac70eb4a09b5bc56973c5d3009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.495119 8883 tablet_service.cc:1467] Processing CreateTablet for tablet 38c779579a0648ee982d552a8fec0bec (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240426 09:53:39.508370 8878 tablet_service.cc:1467] Processing CreateTablet for tablet 431d4df7c67747fcb0812c91964c801b (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240426 09:53:39.515019 9030 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfc67501c0a441d9ac98720108b6f73 (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240426 09:53:39.513268 8882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ee2df31bb64b6a9536f91149a87c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.517886 9030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfc67501c0a441d9ac98720108b6f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.509822 9032 tablet_service.cc:1467] Processing CreateTablet for tablet 38c779579a0648ee982d552a8fec0bec (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240426 09:53:39.517813 8881 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfc67501c0a441d9ac98720108b6f73 (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240426 09:53:39.512012 8879 tablet_service.cc:1467] Processing CreateTablet for tablet 4af8a55ea6e5429093a5ec516bd3f6a0 (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240426 09:53:39.529114 9032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c779579a0648ee982d552a8fec0bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.526422 9029 tablet_service.cc:1467] Processing CreateTablet for tablet 69d609ac70eb4a09b5bc56973c5d3009 (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240426 09:53:39.517822 9031 tablet_service.cc:1467] Processing CreateTablet for tablet 91ee2df31bb64b6a9536f91149a87c0d (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240426 09:53:39.531049 8883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c779579a0648ee982d552a8fec0bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.528574 8877 tablet_service.cc:1467] Processing CreateTablet for tablet df73379416114f8ab7efd3caf9c22e7e (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240426 09:53:39.528653 8875 tablet_service.cc:1467] Processing CreateTablet for tablet 7b463d0d799d4c77abcbb023cc934bf2 (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240426 09:53:39.536070 8876 tablet_service.cc:1467] Processing CreateTablet for tablet 97deaa8654f64ad2a63314ffaa91d955 (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240426 09:53:39.540359 8879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af8a55ea6e5429093a5ec516bd3f6a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.547324 9026 tablet_service.cc:1467] Processing CreateTablet for tablet df73379416114f8ab7efd3caf9c22e7e (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240426 09:53:39.548529 9029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d609ac70eb4a09b5bc56973c5d3009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.548842 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 4af8a55ea6e5429093a5ec516bd3f6a0 (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240426 09:53:39.552795 9027 tablet_service.cc:1467] Processing CreateTablet for tablet 431d4df7c67747fcb0812c91964c801b (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240426 09:53:39.556907 8881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfc67501c0a441d9ac98720108b6f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.562011 9031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ee2df31bb64b6a9536f91149a87c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.562831 8875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b463d0d799d4c77abcbb023cc934bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.559880 9024 tablet_service.cc:1467] Processing CreateTablet for tablet 7b463d0d799d4c77abcbb023cc934bf2 (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240426 09:53:39.565100 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af8a55ea6e5429093a5ec516bd3f6a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.566406 8876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97deaa8654f64ad2a63314ffaa91d955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.559118 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 97deaa8654f64ad2a63314ffaa91d955 (DEFAULT_TABLE table=test-workload [id=e7be83a9e4164f5abc9c5fa45997761f]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240426 09:53:39.573698 8877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df73379416114f8ab7efd3caf9c22e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.576534 8878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431d4df7c67747fcb0812c91964c801b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.578536 9026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df73379416114f8ab7efd3caf9c22e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.583254 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97deaa8654f64ad2a63314ffaa91d955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.595369 9085 tablet_bootstrap.cc:492] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3: No bootstrap required, opened a new log
I20240426 09:53:39.595841 9085 ts_tablet_manager.cc:1367] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3: Time spent bootstrapping tablet: real 0.133s user 0.009s sys 0.010s
I20240426 09:53:39.598024 9024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b463d0d799d4c77abcbb023cc934bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:53:39.605696 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431d4df7c67747fcb0812c91964c801b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 09:53:39.615837 8837 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 09:53:39.620280 9085 raft_consensus.cc:348] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.633323 9085 raft_consensus.cc:374] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:39.633626 9085 raft_consensus.cc:729] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Initialized, Role: FOLLOWER
I20240426 09:53:39.635179 9085 consensus_queue.cc:260] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.674337 8993 heartbeater.cc:503] Master 127.8.141.190:37867 was elected leader, sending a full tablet report...
I20240426 09:53:39.694373 9085 ts_tablet_manager.cc:1398] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3: Time spent starting tablet: real 0.098s user 0.007s sys 0.011s
I20240426 09:53:39.702697 9085 tablet_bootstrap.cc:492] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3: Bootstrap starting.
I20240426 09:53:39.708496 9088 tablet_bootstrap.cc:492] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9: Bootstrap starting.
I20240426 09:53:39.710573 9085 tablet_bootstrap.cc:654] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:39.717167 9088 tablet_bootstrap.cc:654] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:39.722920 9089 tablet_bootstrap.cc:492] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d: Bootstrap starting.
I20240426 09:53:39.726008 9085 tablet_bootstrap.cc:492] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3: No bootstrap required, opened a new log
I20240426 09:53:39.726418 9085 ts_tablet_manager.cc:1367] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.000s
I20240426 09:53:39.728893 9085 raft_consensus.cc:348] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.729532 9089 tablet_bootstrap.cc:654] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:39.729650 9085 raft_consensus.cc:374] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:39.729997 9085 raft_consensus.cc:729] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Initialized, Role: FOLLOWER
I20240426 09:53:39.730865 9085 consensus_queue.cc:260] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.733225 9085 ts_tablet_manager.cc:1398] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240426 09:53:39.734201 9085 tablet_bootstrap.cc:492] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3: Bootstrap starting.
I20240426 09:53:39.738330 9085 tablet_bootstrap.cc:654] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:39.744606 9088 tablet_bootstrap.cc:492] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9: No bootstrap required, opened a new log
I20240426 09:53:39.745102 9088 ts_tablet_manager.cc:1367] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9: Time spent bootstrapping tablet: real 0.045s user 0.011s sys 0.011s
I20240426 09:53:39.748062 9089 tablet_bootstrap.cc:492] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d: No bootstrap required, opened a new log
I20240426 09:53:39.748473 9089 ts_tablet_manager.cc:1367] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.003s
I20240426 09:53:39.750236 9088 raft_consensus.cc:348] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.751008 9088 raft_consensus.cc:374] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:39.751304 9088 raft_consensus.cc:729] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Initialized, Role: FOLLOWER
I20240426 09:53:39.751389 9089 raft_consensus.cc:348] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.752130 9089 raft_consensus.cc:374] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:39.752440 9089 raft_consensus.cc:729] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Initialized, Role: FOLLOWER
I20240426 09:53:39.751993 9088 consensus_queue.cc:260] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.753157 9089 consensus_queue.cc:260] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.754997 9085 tablet_bootstrap.cc:492] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3: No bootstrap required, opened a new log
I20240426 09:53:39.755437 9085 ts_tablet_manager.cc:1367] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.006s
I20240426 09:53:39.757900 9085 raft_consensus.cc:348] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.758606 9085 raft_consensus.cc:374] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:39.758908 9085 raft_consensus.cc:729] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Initialized, Role: FOLLOWER
I20240426 09:53:39.760246 9085 consensus_queue.cc:260] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.762217 9089 ts_tablet_manager.cc:1398] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20240426 09:53:39.762320 9085 ts_tablet_manager.cc:1398] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240426 09:53:39.762727 9067 heartbeater.cc:503] Master 127.8.141.190:37867 was elected leader, sending a full tablet report...
I20240426 09:53:39.763218 9085 tablet_bootstrap.cc:492] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3: Bootstrap starting.
I20240426 09:53:39.767086 9089 tablet_bootstrap.cc:492] T 4af8a55ea6e5429093a5ec516bd3f6a0 P d99df4d00f1e42699f4e40da65bbc33d: Bootstrap starting.
I20240426 09:53:39.768488 9085 tablet_bootstrap.cc:654] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:39.776434 9088 ts_tablet_manager.cc:1398] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9: Time spent starting tablet: real 0.028s user 0.004s sys 0.010s
I20240426 09:53:39.776558 9089 tablet_bootstrap.cc:654] T 4af8a55ea6e5429093a5ec516bd3f6a0 P d99df4d00f1e42699f4e40da65bbc33d: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:39.777139 8918 heartbeater.cc:503] Master 127.8.141.190:37867 was elected leader, sending a full tablet report...
I20240426 09:53:39.778837 9088 tablet_bootstrap.cc:492] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9: Bootstrap starting.
I20240426 09:53:39.779227 9087 raft_consensus.cc:482] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:53:39.779651 9087 raft_consensus.cc:504] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.783236 9087 leader_election.cc:288] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:39.792704 9088 tablet_bootstrap.cc:654] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9: Neither blocks nor log segments found. Creating new log.
W20240426 09:53:39.799333 8994 tablet.cc:2369] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 09:53:39.815061 9085 tablet_bootstrap.cc:492] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3: No bootstrap required, opened a new log
I20240426 09:53:39.815958 9085 ts_tablet_manager.cc:1367] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3: Time spent bootstrapping tablet: real 0.053s user 0.010s sys 0.004s
I20240426 09:53:39.818706 9089 tablet_bootstrap.cc:492] T 4af8a55ea6e5429093a5ec516bd3f6a0 P d99df4d00f1e42699f4e40da65bbc33d: No bootstrap required, opened a new log
I20240426 09:53:39.819161 9089 ts_tablet_manager.cc:1367] T 4af8a55ea6e5429093a5ec516bd3f6a0 P d99df4d00f1e42699f4e40da65bbc33d: Time spent bootstrapping tablet: real 0.052s user 0.014s sys 0.001s
I20240426 09:53:39.819056 9085 raft_consensus.cc:348] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.819845 9085 raft_consensus.cc:374] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:39.820150 9085 raft_consensus.cc:729] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Initialized, Role: FOLLOWER
I20240426 09:53:39.820937 9085 consensus_queue.cc:260] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.822454 9089 raft_consensus.cc:348] T 4af8a55ea6e5429093a5ec516bd3f6a0 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.824640 9089 raft_consensus.cc:374] T 4af8a55ea6e5429093a5ec516bd3f6a0 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:39.825971 9085 ts_tablet_manager.cc:1398] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240426 09:53:39.826828 9085 tablet_bootstrap.cc:492] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3: Bootstrap starting.
I20240426 09:53:39.828121 9088 tablet_bootstrap.cc:492] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9: No bootstrap required, opened a new log
I20240426 09:53:39.830353 9088 ts_tablet_manager.cc:1367] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9: Time spent bootstrapping tablet: real 0.053s user 0.014s sys 0.003s
I20240426 09:53:39.825973 9089 raft_consensus.cc:729] T 4af8a55ea6e5429093a5ec516bd3f6a0 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Initialized, Role: FOLLOWER
I20240426 09:53:39.830435 8893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ee2df31bb64b6a9536f91149a87c0d" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbe92f5959a47dda817bbec2148eab9" is_pre_election: true
I20240426 09:53:39.831296 9089 consensus_queue.cc:260] T 4af8a55ea6e5429093a5ec516bd3f6a0 P d99df4d00f1e42699f4e40da65bbc33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
W20240426 09:53:39.833001 8933 leader_election.cc:341] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:53:39.833266 9089 ts_tablet_manager.cc:1398] T 4af8a55ea6e5429093a5ec516bd3f6a0 P d99df4d00f1e42699f4e40da65bbc33d: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20240426 09:53:39.835633 9088 raft_consensus.cc:348] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.836143 9088 raft_consensus.cc:374] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:39.836359 9088 raft_consensus.cc:729] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Initialized, Role: FOLLOWER
I20240426 09:53:39.836864 9088 consensus_queue.cc:260] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.838443 9088 ts_tablet_manager.cc:1398] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240426 09:53:39.839133 9088 tablet_bootstrap.cc:492] T 7b463d0d799d4c77abcbb023cc934bf2 P cdbe92f5959a47dda817bbec2148eab9: Bootstrap starting.
I20240426 09:53:39.840723 9085 tablet_bootstrap.cc:654] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:39.841945 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ee2df31bb64b6a9536f91149a87c0d" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d" is_pre_election: true
I20240426 09:53:39.842446 9089 tablet_bootstrap.cc:492] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d: Bootstrap starting.
W20240426 09:53:39.843652 8934 leader_election.cc:341] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:53:39.844064 8934 leader_election.cc:302] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3; no voters: cdbe92f5959a47dda817bbec2148eab9, d99df4d00f1e42699f4e40da65bbc33d
I20240426 09:53:39.844761 9087 raft_consensus.cc:2726] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:53:39.849234 9089 tablet_bootstrap.cc:654] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:39.860946 9085 tablet_bootstrap.cc:492] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3: No bootstrap required, opened a new log
I20240426 09:53:39.862151 9089 tablet_bootstrap.cc:492] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d: No bootstrap required, opened a new log
I20240426 09:53:39.862592 9089 ts_tablet_manager.cc:1367] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20240426 09:53:39.863480 9088 tablet_bootstrap.cc:654] T 7b463d0d799d4c77abcbb023cc934bf2 P cdbe92f5959a47dda817bbec2148eab9: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:39.862223 9085 ts_tablet_manager.cc:1367] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3: Time spent bootstrapping tablet: real 0.036s user 0.013s sys 0.004s
I20240426 09:53:39.866430 9089 raft_consensus.cc:348] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.866514 9085 raft_consensus.cc:348] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.867166 9089 raft_consensus.cc:374] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:39.867286 9085 raft_consensus.cc:374] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:39.867480 9089 raft_consensus.cc:729] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Initialized, Role: FOLLOWER
I20240426 09:53:39.867622 9085 raft_consensus.cc:729] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Initialized, Role: FOLLOWER
I20240426 09:53:39.868116 9089 consensus_queue.cc:260] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.869745 9089 ts_tablet_manager.cc:1398] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240426 09:53:39.870498 9089 tablet_bootstrap.cc:492] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d: Bootstrap starting.
I20240426 09:53:39.870527 9095 raft_consensus.cc:482] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:53:39.868790 9085 consensus_queue.cc:260] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.871054 9095 raft_consensus.cc:504] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.872598 9085 ts_tablet_manager.cc:1398] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240426 09:53:39.873894 9095 leader_election.cc:288] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991)
I20240426 09:53:39.873409 9085 tablet_bootstrap.cc:492] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3: Bootstrap starting.
I20240426 09:53:39.880803 9089 tablet_bootstrap.cc:654] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:39.881927 9085 tablet_bootstrap.cc:654] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:39.883152 9088 tablet_bootstrap.cc:492] T 7b463d0d799d4c77abcbb023cc934bf2 P cdbe92f5959a47dda817bbec2148eab9: No bootstrap required, opened a new log
I20240426 09:53:39.883587 9088 ts_tablet_manager.cc:1367] T 7b463d0d799d4c77abcbb023cc934bf2 P cdbe92f5959a47dda817bbec2148eab9: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.000s
I20240426 09:53:39.886078 9088 raft_consensus.cc:348] T 7b463d0d799d4c77abcbb023cc934bf2 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.886734 9088 raft_consensus.cc:374] T 7b463d0d799d4c77abcbb023cc934bf2 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:39.887022 9088 raft_consensus.cc:729] T 7b463d0d799d4c77abcbb023cc934bf2 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Initialized, Role: FOLLOWER
I20240426 09:53:39.889600 9088 consensus_queue.cc:260] T 7b463d0d799d4c77abcbb023cc934bf2 P cdbe92f5959a47dda817bbec2148eab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.891880 9088 ts_tablet_manager.cc:1398] T 7b463d0d799d4c77abcbb023cc934bf2 P cdbe92f5959a47dda817bbec2148eab9: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240426 09:53:39.898464 9088 tablet_bootstrap.cc:492] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9: Bootstrap starting.
I20240426 09:53:39.904956 9088 tablet_bootstrap.cc:654] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:39.913173 9089 tablet_bootstrap.cc:492] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d: No bootstrap required, opened a new log
I20240426 09:53:39.913650 9089 ts_tablet_manager.cc:1367] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d: Time spent bootstrapping tablet: real 0.043s user 0.013s sys 0.003s
I20240426 09:53:39.916704 9089 raft_consensus.cc:348] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.917726 9089 raft_consensus.cc:374] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:39.917995 9089 raft_consensus.cc:729] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Initialized, Role: FOLLOWER
I20240426 09:53:39.918709 9089 consensus_queue.cc:260] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.920542 8893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b463d0d799d4c77abcbb023cc934bf2" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbe92f5959a47dda817bbec2148eab9" is_pre_election: true
I20240426 09:53:39.921751 8893 raft_consensus.cc:2454] T 7b463d0d799d4c77abcbb023cc934bf2 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 0.
I20240426 09:53:39.927341 9085 tablet_bootstrap.cc:492] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3: No bootstrap required, opened a new log
I20240426 09:53:39.928073 9007 leader_election.cc:302] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdbe92f5959a47dda817bbec2148eab9, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:39.928386 9085 ts_tablet_manager.cc:1367] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3: Time spent bootstrapping tablet: real 0.055s user 0.027s sys 0.016s
I20240426 09:53:39.929476 9095 raft_consensus.cc:2781] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:53:39.929839 9095 raft_consensus.cc:482] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:53:39.930158 9095 raft_consensus.cc:3037] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:39.928642 9089 ts_tablet_manager.cc:1398] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20240426 09:53:39.931830 9089 tablet_bootstrap.cc:492] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d: Bootstrap starting.
I20240426 09:53:39.932193 9085 raft_consensus.cc:348] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.932623 9087 raft_consensus.cc:482] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:53:39.933851 9087 raft_consensus.cc:504] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.935640 9087 leader_election.cc:288] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:39.936353 9085 raft_consensus.cc:374] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:39.936452 8893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbe92f5959a47dda817bbec2148eab9" is_pre_election: true
I20240426 09:53:39.936702 9085 raft_consensus.cc:729] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Initialized, Role: FOLLOWER
W20240426 09:53:39.938110 8933 leader_election.cc:341] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:53:39.938939 9089 tablet_bootstrap.cc:654] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:39.940936 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d" is_pre_election: true
W20240426 09:53:39.942289 8934 leader_election.cc:341] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:53:39.941958 9085 consensus_queue.cc:260] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.942728 8934 leader_election.cc:302] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3; no voters: cdbe92f5959a47dda817bbec2148eab9, d99df4d00f1e42699f4e40da65bbc33d
I20240426 09:53:39.943845 9087 raft_consensus.cc:2726] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:53:39.945204 9095 raft_consensus.cc:504] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.946969 9095 leader_election.cc:288] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991)
I20240426 09:53:39.947652 9085 ts_tablet_manager.cc:1398] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3: Time spent starting tablet: real 0.018s user 0.002s sys 0.004s
I20240426 09:53:39.948472 9085 tablet_bootstrap.cc:492] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3: Bootstrap starting.
I20240426 09:53:39.949736 9088 tablet_bootstrap.cc:492] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9: No bootstrap required, opened a new log
I20240426 09:53:39.949637 8893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b463d0d799d4c77abcbb023cc934bf2" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
I20240426 09:53:39.950196 8893 raft_consensus.cc:3037] T 7b463d0d799d4c77abcbb023cc934bf2 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:39.950172 9088 ts_tablet_manager.cc:1367] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.001s
I20240426 09:53:39.958500 9085 tablet_bootstrap.cc:654] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:39.963994 8893 raft_consensus.cc:2454] T 7b463d0d799d4c77abcbb023cc934bf2 P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 1.
I20240426 09:53:39.965353 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b463d0d799d4c77abcbb023cc934bf2" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3" is_pre_election: true
I20240426 09:53:39.966821 8967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b463d0d799d4c77abcbb023cc934bf2" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:53:39.967988 9007 leader_election.cc:302] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdbe92f5959a47dda817bbec2148eab9, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:39.968765 9095 raft_consensus.cc:2781] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:53:39.968972 9088 raft_consensus.cc:348] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
W20240426 09:53:39.970852 9007 leader_election.cc:341] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991): Illegal state: must be running to vote when last-logged opid is not known
W20240426 09:53:39.975548 9007 leader_election.cc:341] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:53:39.971875 9088 raft_consensus.cc:374] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:39.976930 9088 raft_consensus.cc:729] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Initialized, Role: FOLLOWER
I20240426 09:53:39.977475 9088 consensus_queue.cc:260] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.979974 9088 ts_tablet_manager.cc:1398] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9: Time spent starting tablet: real 0.021s user 0.004s sys 0.002s
I20240426 09:53:39.980862 9088 tablet_bootstrap.cc:492] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9: Bootstrap starting.
I20240426 09:53:39.986941 9095 raft_consensus.cc:686] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [term 1 LEADER]: Becoming Leader. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Running, Role: LEADER
I20240426 09:53:39.988821 9095 consensus_queue.cc:237] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.991503 9089 tablet_bootstrap.cc:492] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d: No bootstrap required, opened a new log
I20240426 09:53:39.992043 9089 ts_tablet_manager.cc:1367] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d: Time spent bootstrapping tablet: real 0.060s user 0.018s sys 0.011s
I20240426 09:53:39.995100 9089 raft_consensus.cc:348] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.996518 9089 raft_consensus.cc:374] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:39.996870 9089 raft_consensus.cc:729] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Initialized, Role: FOLLOWER
I20240426 09:53:39.997594 9089 consensus_queue.cc:260] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:39.999821 9089 ts_tablet_manager.cc:1398] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20240426 09:53:40.001165 9089 tablet_bootstrap.cc:492] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d: Bootstrap starting.
I20240426 09:53:40.003136 9088 tablet_bootstrap.cc:654] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:40.004101 8790 catalog_manager.cc:5484] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d reported cstate change: term changed from 0 to 1, leader changed from <none> to d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d99df4d00f1e42699f4e40da65bbc33d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: HEALTHY } } }
I20240426 09:53:40.007079 9089 tablet_bootstrap.cc:654] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:40.010455 9085 tablet_bootstrap.cc:492] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3: No bootstrap required, opened a new log
I20240426 09:53:40.011049 9085 ts_tablet_manager.cc:1367] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3: Time spent bootstrapping tablet: real 0.063s user 0.014s sys 0.004s
I20240426 09:53:40.015905 9085 raft_consensus.cc:348] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.016651 9085 raft_consensus.cc:374] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:40.016939 9085 raft_consensus.cc:729] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Initialized, Role: FOLLOWER
I20240426 09:53:40.019289 9085 consensus_queue.cc:260] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.025110 9085 ts_tablet_manager.cc:1398] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20240426 09:53:40.025969 9085 tablet_bootstrap.cc:492] T 7b463d0d799d4c77abcbb023cc934bf2 P 9e98a77341c74d6d90d01cd3e39586b3: Bootstrap starting.
I20240426 09:53:40.026961 9088 tablet_bootstrap.cc:492] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9: No bootstrap required, opened a new log
I20240426 09:53:40.028059 9110 raft_consensus.cc:482] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:53:40.027456 9088 ts_tablet_manager.cc:1367] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.013s
I20240426 09:53:40.028524 9110 raft_consensus.cc:504] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.031231 9110 leader_election.cc:288] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.032495 9085 tablet_bootstrap.cc:654] T 7b463d0d799d4c77abcbb023cc934bf2 P 9e98a77341c74d6d90d01cd3e39586b3: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:40.032645 8893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97deaa8654f64ad2a63314ffaa91d955" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbe92f5959a47dda817bbec2148eab9" is_pre_election: true
W20240426 09:53:40.033778 8933 leader_election.cc:341] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:53:40.033963 9088 raft_consensus.cc:348] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.032487 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97deaa8654f64ad2a63314ffaa91d955" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d" is_pre_election: true
W20240426 09:53:40.036324 8934 leader_election.cc:341] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:53:40.037021 8934 leader_election.cc:302] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3; no voters: cdbe92f5959a47dda817bbec2148eab9, d99df4d00f1e42699f4e40da65bbc33d
I20240426 09:53:40.038381 9110 raft_consensus.cc:2726] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:53:40.034647 9088 raft_consensus.cc:374] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:40.044837 9088 raft_consensus.cc:729] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Initialized, Role: FOLLOWER
I20240426 09:53:40.046411 9088 consensus_queue.cc:260] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.048903 9088 ts_tablet_manager.cc:1398] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9: Time spent starting tablet: real 0.020s user 0.002s sys 0.003s
I20240426 09:53:40.049918 9088 tablet_bootstrap.cc:492] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9: Bootstrap starting.
I20240426 09:53:40.052587 9089 tablet_bootstrap.cc:492] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d: No bootstrap required, opened a new log
I20240426 09:53:40.053040 9089 ts_tablet_manager.cc:1367] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.014s
I20240426 09:53:40.055649 9110 raft_consensus.cc:482] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:53:40.056083 9110 raft_consensus.cc:504] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.056856 9088 tablet_bootstrap.cc:654] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:40.058320 9110 leader_election.cc:288] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.059705 8893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbe92f5959a47dda817bbec2148eab9" is_pre_election: true
I20240426 09:53:40.060356 8893 raft_consensus.cc:2454] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 0.
I20240426 09:53:40.062026 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d" is_pre_election: true
I20240426 09:53:40.062644 9042 raft_consensus.cc:2454] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 0.
I20240426 09:53:40.062976 9085 tablet_bootstrap.cc:492] T 7b463d0d799d4c77abcbb023cc934bf2 P 9e98a77341c74d6d90d01cd3e39586b3: No bootstrap required, opened a new log
I20240426 09:53:40.063385 9085 ts_tablet_manager.cc:1367] T 7b463d0d799d4c77abcbb023cc934bf2 P 9e98a77341c74d6d90d01cd3e39586b3: Time spent bootstrapping tablet: real 0.038s user 0.021s sys 0.011s
I20240426 09:53:40.063951 8933 leader_election.cc:302] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, cdbe92f5959a47dda817bbec2148eab9; no voters:
I20240426 09:53:40.065699 9110 raft_consensus.cc:2781] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:53:40.065661 9085 raft_consensus.cc:348] T 7b463d0d799d4c77abcbb023cc934bf2 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.066232 9110 raft_consensus.cc:482] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:53:40.066401 9085 raft_consensus.cc:374] T 7b463d0d799d4c77abcbb023cc934bf2 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:40.066851 9110 raft_consensus.cc:3037] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.067391 9085 raft_consensus.cc:729] T 7b463d0d799d4c77abcbb023cc934bf2 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Initialized, Role: FOLLOWER
I20240426 09:53:40.068006 9085 consensus_queue.cc:260] T 7b463d0d799d4c77abcbb023cc934bf2 P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.055641 9089 raft_consensus.cc:348] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.070101 9089 raft_consensus.cc:374] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:40.070422 9089 raft_consensus.cc:729] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Initialized, Role: FOLLOWER
I20240426 09:53:40.071209 9089 consensus_queue.cc:260] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.071702 9085 ts_tablet_manager.cc:1398] T 7b463d0d799d4c77abcbb023cc934bf2 P 9e98a77341c74d6d90d01cd3e39586b3: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20240426 09:53:40.073848 9085 tablet_bootstrap.cc:492] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3: Bootstrap starting.
I20240426 09:53:40.073810 9089 ts_tablet_manager.cc:1398] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20240426 09:53:40.075021 9089 tablet_bootstrap.cc:492] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d: Bootstrap starting.
I20240426 09:53:40.075503 9110 raft_consensus.cc:504] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.079216 9088 tablet_bootstrap.cc:492] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9: No bootstrap required, opened a new log
I20240426 09:53:40.081100 9085 tablet_bootstrap.cc:654] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:40.080740 9088 ts_tablet_manager.cc:1367] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.013s
I20240426 09:53:40.084051 9110 leader_election.cc:288] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.085664 8893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
I20240426 09:53:40.085844 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:40.086297 8893 raft_consensus.cc:3037] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.086580 9042 raft_consensus.cc:3037] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.087807 9088 raft_consensus.cc:348] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.088655 9088 raft_consensus.cc:374] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:40.088975 9088 raft_consensus.cc:729] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Initialized, Role: FOLLOWER
I20240426 09:53:40.089648 9088 consensus_queue.cc:260] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.094640 8893 raft_consensus.cc:2454] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 1.
I20240426 09:53:40.095836 9089 tablet_bootstrap.cc:654] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:40.095891 9110 raft_consensus.cc:482] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:53:40.096459 9110 raft_consensus.cc:504] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.098747 9110 leader_election.cc:288] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.099195 9110 raft_consensus.cc:482] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:53:40.099577 9110 raft_consensus.cc:504] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.100068 8933 leader_election.cc:302] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, cdbe92f5959a47dda817bbec2148eab9; no voters:
I20240426 09:53:40.100900 9087 raft_consensus.cc:2781] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:53:40.101945 9042 raft_consensus.cc:2454] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 1.
I20240426 09:53:40.106230 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8a55ea6e5429093a5ec516bd3f6a0" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d" is_pre_election: true
I20240426 09:53:40.109371 9088 ts_tablet_manager.cc:1398] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20240426 09:53:40.109599 9042 raft_consensus.cc:2454] T 4af8a55ea6e5429093a5ec516bd3f6a0 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 0.
I20240426 09:53:40.113371 9110 leader_election.cc:288] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.117399 9088 tablet_bootstrap.cc:492] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9: Bootstrap starting.
I20240426 09:53:40.111843 8892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8a55ea6e5429093a5ec516bd3f6a0" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbe92f5959a47dda817bbec2148eab9" is_pre_election: true
I20240426 09:53:40.117343 9087 raft_consensus.cc:686] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 LEADER]: Becoming Leader. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:40.119657 8893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbe92f5959a47dda817bbec2148eab9" is_pre_election: true
I20240426 09:53:40.119252 8934 leader_election.cc:302] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:40.123540 9094 raft_consensus.cc:482] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:53:40.125726 8933 leader_election.cc:341] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:53:40.124751 9087 consensus_queue.cc:237] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.124814 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d" is_pre_election: true
W20240426 09:53:40.128422 8933 leader_election.cc:341] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:53:40.126201 9094 raft_consensus.cc:504] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.127642 9110 raft_consensus.cc:2781] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:53:40.129086 9042 raft_consensus.cc:2454] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 0.
I20240426 09:53:40.129657 9110 raft_consensus.cc:482] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:53:40.131803 9110 raft_consensus.cc:3037] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.132361 8934 leader_election.cc:302] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters: cdbe92f5959a47dda817bbec2148eab9
I20240426 09:53:40.136610 9085 tablet_bootstrap.cc:492] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3: No bootstrap required, opened a new log
I20240426 09:53:40.160944 9089 tablet_bootstrap.cc:492] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d: No bootstrap required, opened a new log
I20240426 09:53:40.168372 9085 ts_tablet_manager.cc:1367] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3: Time spent bootstrapping tablet: real 0.095s user 0.035s sys 0.010s
I20240426 09:53:40.135566 9124 raft_consensus.cc:2781] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:53:40.168839 9089 ts_tablet_manager.cc:1367] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d: Time spent bootstrapping tablet: real 0.094s user 0.029s sys 0.022s
I20240426 09:53:40.138293 9094 leader_election.cc:288] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.148655 9095 raft_consensus.cc:482] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:53:40.172078 9095 raft_consensus.cc:504] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.171491 9124 raft_consensus.cc:482] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:53:40.176132 9110 raft_consensus.cc:504] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.176493 8967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97deaa8654f64ad2a63314ffaa91d955" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3" is_pre_election: true
I20240426 09:53:40.177008 8892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97deaa8654f64ad2a63314ffaa91d955" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbe92f5959a47dda817bbec2148eab9" is_pre_election: true
I20240426 09:53:40.177934 8967 raft_consensus.cc:2454] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 0.
I20240426 09:53:40.178879 9085 raft_consensus.cc:348] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.180974 9085 raft_consensus.cc:374] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:40.174991 9124 raft_consensus.cc:3037] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.181824 9007 leader_election.cc:302] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:40.181298 9085 raft_consensus.cc:729] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Initialized, Role: FOLLOWER
I20240426 09:53:40.182758 9085 consensus_queue.cc:260] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.177469 9095 leader_election.cc:288] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991)
I20240426 09:53:40.183902 8892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8a55ea6e5429093a5ec516bd3f6a0" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
W20240426 09:53:40.185055 9007 leader_election.cc:341] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:53:40.185077 9112 raft_consensus.cc:2781] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:53:40.185849 9110 leader_election.cc:288] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.185870 9112 raft_consensus.cc:482] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 09:53:40.186085 8933 leader_election.cc:341] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:53:40.186439 9112 raft_consensus.cc:3037] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.190224 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8a55ea6e5429093a5ec516bd3f6a0" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:40.191439 9088 tablet_bootstrap.cc:654] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:40.191612 9042 raft_consensus.cc:3037] T 4af8a55ea6e5429093a5ec516bd3f6a0 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.193210 9089 raft_consensus.cc:348] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.195729 9089 raft_consensus.cc:374] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:40.196091 9089 raft_consensus.cc:729] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Initialized, Role: FOLLOWER
I20240426 09:53:40.202340 9085 ts_tablet_manager.cc:1398] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3: Time spent starting tablet: real 0.034s user 0.009s sys 0.003s
I20240426 09:53:40.195678 8788 catalog_manager.cc:5484] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130). New cstate: current_term: 1 leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:40.203886 9089 consensus_queue.cc:260] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.206933 9089 ts_tablet_manager.cc:1398] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20240426 09:53:40.207963 9089 tablet_bootstrap.cc:492] T 91ee2df31bb64b6a9536f91149a87c0d P d99df4d00f1e42699f4e40da65bbc33d: Bootstrap starting.
I20240426 09:53:40.208186 9042 raft_consensus.cc:2454] T 4af8a55ea6e5429093a5ec516bd3f6a0 P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 1.
I20240426 09:53:40.210897 9094 raft_consensus.cc:482] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:53:40.211321 9094 raft_consensus.cc:504] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.215027 8934 leader_election.cc:302] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters: cdbe92f5959a47dda817bbec2148eab9
I20240426 09:53:40.215878 9087 raft_consensus.cc:2781] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:53:40.216408 9087 raft_consensus.cc:686] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 LEADER]: Becoming Leader. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:40.217187 9087 consensus_queue.cc:237] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.220396 9094 leader_election.cc:288] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.213603 9112 raft_consensus.cc:504] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.224720 9112 leader_election.cc:288] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991)
I20240426 09:53:40.224757 8892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97deaa8654f64ad2a63314ffaa91d955" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
I20240426 09:53:40.226148 8967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97deaa8654f64ad2a63314ffaa91d955" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:53:40.226110 9124 raft_consensus.cc:504] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.226974 8967 raft_consensus.cc:3037] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Advancing to term 1
W20240426 09:53:40.227010 9007 leader_election.cc:341] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:53:40.229722 9124 leader_election.cc:288] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.230921 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:40.231542 9042 raft_consensus.cc:3037] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.233078 9089 tablet_bootstrap.cc:654] T 91ee2df31bb64b6a9536f91149a87c0d P d99df4d00f1e42699f4e40da65bbc33d: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:40.232942 8892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
W20240426 09:53:40.234344 8933 leader_election.cc:341] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:53:40.240818 8967 raft_consensus.cc:2454] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 1.
I20240426 09:53:40.241778 9007 leader_election.cc:302] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters: cdbe92f5959a47dda817bbec2148eab9
I20240426 09:53:40.242797 9112 raft_consensus.cc:2781] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:53:40.243571 9112 raft_consensus.cc:686] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 1 LEADER]: Becoming Leader. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Running, Role: LEADER
I20240426 09:53:40.244359 9112 consensus_queue.cc:237] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.245469 9041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df73379416114f8ab7efd3caf9c22e7e" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d" is_pre_election: true
I20240426 09:53:40.245844 9040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ee2df31bb64b6a9536f91149a87c0d" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d" is_pre_election: true
I20240426 09:53:40.246174 9041 raft_consensus.cc:2454] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 0.
W20240426 09:53:40.247437 8859 leader_election.cc:341] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:53:40.247682 9088 tablet_bootstrap.cc:492] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9: No bootstrap required, opened a new log
I20240426 09:53:40.248242 9088 ts_tablet_manager.cc:1367] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9: Time spent bootstrapping tablet: real 0.133s user 0.012s sys 0.007s
I20240426 09:53:40.248311 8859 leader_election.cc:302] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdbe92f5959a47dda817bbec2148eab9, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:40.249189 9094 raft_consensus.cc:2781] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:53:40.249518 9094 raft_consensus.cc:482] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:53:40.249848 9094 raft_consensus.cc:3037] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.249006 8787 catalog_manager.cc:5484] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130). New cstate: current_term: 1 leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:40.250864 9088 raft_consensus.cc:348] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.251672 9088 raft_consensus.cc:374] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:40.251889 9088 raft_consensus.cc:729] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Initialized, Role: FOLLOWER
I20240426 09:53:40.252355 9088 consensus_queue.cc:260] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.254698 9042 raft_consensus.cc:2454] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 1.
I20240426 09:53:40.255795 8934 leader_election.cc:302] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters: cdbe92f5959a47dda817bbec2148eab9
I20240426 09:53:40.259371 9094 raft_consensus.cc:504] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.260897 8788 catalog_manager.cc:5484] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d reported cstate change: term changed from 0 to 1, leader changed from <none> to d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d99df4d00f1e42699f4e40da65bbc33d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: HEALTHY } } }
I20240426 09:53:40.260170 9087 raft_consensus.cc:2781] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:53:40.262557 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df73379416114f8ab7efd3caf9c22e7e" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:40.262569 9087 raft_consensus.cc:686] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 LEADER]: Becoming Leader. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:40.263200 9042 raft_consensus.cc:3037] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.263904 9087 consensus_queue.cc:237] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.268656 9094 leader_election.cc:288] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 election: Requested vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.271592 9094 raft_consensus.cc:482] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:53:40.272025 9094 raft_consensus.cc:504] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.277459 8967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df73379416114f8ab7efd3caf9c22e7e" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3" is_pre_election: true
I20240426 09:53:40.278127 8967 raft_consensus.cc:2454] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 0.
I20240426 09:53:40.278087 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ee2df31bb64b6a9536f91149a87c0d" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3" is_pre_election: true
I20240426 09:53:40.281631 8966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df73379416114f8ab7efd3caf9c22e7e" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:53:40.282305 8966 raft_consensus.cc:3037] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.289011 8966 raft_consensus.cc:2454] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 1.
I20240426 09:53:40.279703 8968 raft_consensus.cc:2454] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 0.
I20240426 09:53:40.293692 9088 ts_tablet_manager.cc:1398] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9: Time spent starting tablet: real 0.045s user 0.005s sys 0.013s
I20240426 09:53:40.294243 9089 tablet_bootstrap.cc:492] T 91ee2df31bb64b6a9536f91149a87c0d P d99df4d00f1e42699f4e40da65bbc33d: No bootstrap required, opened a new log
I20240426 09:53:40.299304 8787 catalog_manager.cc:5484] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130). New cstate: current_term: 1 leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:40.301378 8858 leader_election.cc:302] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, cdbe92f5959a47dda817bbec2148eab9; no voters:
I20240426 09:53:40.301956 9089 ts_tablet_manager.cc:1367] T 91ee2df31bb64b6a9536f91149a87c0d P d99df4d00f1e42699f4e40da65bbc33d: Time spent bootstrapping tablet: real 0.094s user 0.014s sys 0.015s
I20240426 09:53:40.300112 9042 raft_consensus.cc:2454] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 1.
I20240426 09:53:40.302270 8858 leader_election.cc:302] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, cdbe92f5959a47dda817bbec2148eab9; no voters: d99df4d00f1e42699f4e40da65bbc33d
I20240426 09:53:40.303589 9134 raft_consensus.cc:2781] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:53:40.303671 9133 raft_consensus.cc:2781] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:53:40.304085 9133 raft_consensus.cc:482] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:53:40.304428 9133 raft_consensus.cc:3037] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.306634 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431d4df7c67747fcb0812c91964c801b" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3" is_pre_election: true
I20240426 09:53:40.307240 8968 raft_consensus.cc:2454] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 0.
I20240426 09:53:40.308230 8858 leader_election.cc:302] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, cdbe92f5959a47dda817bbec2148eab9; no voters:
I20240426 09:53:40.301380 9088 tablet_bootstrap.cc:492] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9: Bootstrap starting.
I20240426 09:53:40.310593 9094 leader_election.cc:288] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.312741 9094 raft_consensus.cc:2781] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:53:40.313089 9094 raft_consensus.cc:482] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:53:40.313452 9094 raft_consensus.cc:3037] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.317662 9133 raft_consensus.cc:504] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.318365 9088 tablet_bootstrap.cc:654] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:40.318631 9089 raft_consensus.cc:348] T 91ee2df31bb64b6a9536f91149a87c0d P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.319392 9089 raft_consensus.cc:374] T 91ee2df31bb64b6a9536f91149a87c0d P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:40.320159 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431d4df7c67747fcb0812c91964c801b" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d" is_pre_election: true
I20240426 09:53:40.320163 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ee2df31bb64b6a9536f91149a87c0d" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:53:40.321188 9089 raft_consensus.cc:729] T 91ee2df31bb64b6a9536f91149a87c0d P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Initialized, Role: FOLLOWER
I20240426 09:53:40.320534 9041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ee2df31bb64b6a9536f91149a87c0d" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:40.319615 9133 leader_election.cc:288] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 election: Requested vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.321300 8968 raft_consensus.cc:3037] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.320309 9134 raft_consensus.cc:686] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 1 LEADER]: Becoming Leader. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Running, Role: LEADER
I20240426 09:53:40.322471 9089 consensus_queue.cc:260] T 91ee2df31bb64b6a9536f91149a87c0d P d99df4d00f1e42699f4e40da65bbc33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
W20240426 09:53:40.321906 8859 leader_election.cc:341] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:53:40.323323 9041 raft_consensus.cc:3037] T 91ee2df31bb64b6a9536f91149a87c0d P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.324821 9089 ts_tablet_manager.cc:1398] T 91ee2df31bb64b6a9536f91149a87c0d P d99df4d00f1e42699f4e40da65bbc33d: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20240426 09:53:40.326094 9089 tablet_bootstrap.cc:492] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d: Bootstrap starting.
I20240426 09:53:40.328557 8968 raft_consensus.cc:2454] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 1.
I20240426 09:53:40.329560 8858 leader_election.cc:302] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, cdbe92f5959a47dda817bbec2148eab9; no voters:
I20240426 09:53:40.330381 9136 raft_consensus.cc:2781] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:53:40.330408 9041 raft_consensus.cc:2454] T 91ee2df31bb64b6a9536f91149a87c0d P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 1.
I20240426 09:53:40.331033 9136 raft_consensus.cc:686] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 1 LEADER]: Becoming Leader. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Running, Role: LEADER
I20240426 09:53:40.331944 9136 consensus_queue.cc:237] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.331512 9134 consensus_queue.cc:237] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.333174 9094 raft_consensus.cc:504] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.335994 9089 tablet_bootstrap.cc:654] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:40.336099 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431d4df7c67747fcb0812c91964c801b" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:53:40.337412 8968 raft_consensus.cc:3037] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.337750 9094 leader_election.cc:288] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 election: Requested vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.345350 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431d4df7c67747fcb0812c91964c801b" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
W20240426 09:53:40.346315 8859 leader_election.cc:341] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:53:40.347049 9112 raft_consensus.cc:482] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:53:40.347512 9112 raft_consensus.cc:504] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.349611 9112 leader_election.cc:288] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507)
I20240426 09:53:40.350641 8968 raft_consensus.cc:2454] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 1.
I20240426 09:53:40.350970 9089 tablet_bootstrap.cc:492] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d: No bootstrap required, opened a new log
I20240426 09:53:40.351353 9089 ts_tablet_manager.cc:1367] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.002s
I20240426 09:53:40.352003 8858 leader_election.cc:302] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, cdbe92f5959a47dda817bbec2148eab9; no voters: d99df4d00f1e42699f4e40da65bbc33d
I20240426 09:53:40.352042 8966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3" is_pre_election: true
I20240426 09:53:40.352783 8966 raft_consensus.cc:2454] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 0.
I20240426 09:53:40.353044 9136 raft_consensus.cc:2781] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:53:40.353478 9136 raft_consensus.cc:686] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 1 LEADER]: Becoming Leader. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Running, Role: LEADER
I20240426 09:53:40.353785 9007 leader_election.cc:302] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:40.353690 9089 raft_consensus.cc:348] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.354535 9112 raft_consensus.cc:2781] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:53:40.354337 9136 consensus_queue.cc:237] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.354944 9112 raft_consensus.cc:482] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:53:40.355332 9112 raft_consensus.cc:3037] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.352838 8892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbe92f5959a47dda817bbec2148eab9" is_pre_election: true
I20240426 09:53:40.356072 8892 raft_consensus.cc:2454] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 0.
I20240426 09:53:40.357384 8787 catalog_manager.cc:5484] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "cdbe92f5959a47dda817bbec2148eab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:40.354707 9088 tablet_bootstrap.cc:492] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9: No bootstrap required, opened a new log
I20240426 09:53:40.359830 9088 ts_tablet_manager.cc:1367] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.005s
I20240426 09:53:40.361765 9112 raft_consensus.cc:504] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.364146 9112 leader_election.cc:288] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 election: Requested vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507)
I20240426 09:53:40.364557 9089 raft_consensus.cc:374] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:40.364845 9089 raft_consensus.cc:729] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Initialized, Role: FOLLOWER
I20240426 09:53:40.365429 9089 consensus_queue.cc:260] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.365815 8892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
I20240426 09:53:40.366384 8892 raft_consensus.cc:3037] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.362679 9088 raft_consensus.cc:348] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.366943 9088 raft_consensus.cc:374] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:40.367218 9088 raft_consensus.cc:729] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Initialized, Role: FOLLOWER
I20240426 09:53:40.367336 9089 ts_tablet_manager.cc:1398] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20240426 09:53:40.367801 9088 consensus_queue.cc:260] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.369235 9088 ts_tablet_manager.cc:1398] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240426 09:53:40.369804 9088 tablet_bootstrap.cc:492] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9: Bootstrap starting.
I20240426 09:53:40.371133 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:53:40.371704 8968 raft_consensus.cc:3037] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.374186 9088 tablet_bootstrap.cc:654] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9: Neither blocks nor log segments found. Creating new log.
I20240426 09:53:40.380779 8788 catalog_manager.cc:5484] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "cdbe92f5959a47dda817bbec2148eab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:40.382500 8788 catalog_manager.cc:5484] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "cdbe92f5959a47dda817bbec2148eab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:40.384686 8892 raft_consensus.cc:2454] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 1.
I20240426 09:53:40.385787 9007 leader_election.cc:302] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdbe92f5959a47dda817bbec2148eab9, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:40.386546 9112 raft_consensus.cc:2781] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:53:40.387192 9112 raft_consensus.cc:686] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 1 LEADER]: Becoming Leader. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Running, Role: LEADER
I20240426 09:53:40.388062 9112 consensus_queue.cc:237] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.389768 9088 tablet_bootstrap.cc:492] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9: No bootstrap required, opened a new log
I20240426 09:53:40.390172 9088 ts_tablet_manager.cc:1367] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.007s
I20240426 09:53:40.390735 8968 raft_consensus.cc:2454] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 1.
I20240426 09:53:40.392534 9088 raft_consensus.cc:348] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.393185 9088 raft_consensus.cc:374] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:53:40.393429 9088 raft_consensus.cc:729] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Initialized, Role: FOLLOWER
I20240426 09:53:40.393965 9088 consensus_queue.cc:260] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.396054 9088 ts_tablet_manager.cc:1398] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240426 09:53:40.398928 8788 catalog_manager.cc:5484] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d reported cstate change: term changed from 0 to 1, leader changed from <none> to d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d99df4d00f1e42699f4e40da65bbc33d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: HEALTHY } } }
I20240426 09:53:40.512748 8968 raft_consensus.cc:3037] T 7b463d0d799d4c77abcbb023cc934bf2 P 9e98a77341c74d6d90d01cd3e39586b3 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.520808 9112 consensus_queue.cc:1035] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:40.539345 9112 consensus_queue.cc:1035] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:40.544742 9134 raft_consensus.cc:482] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:53:40.545305 9134 raft_consensus.cc:504] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.547358 9134 leader_election.cc:288] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.548918 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8a55ea6e5429093a5ec516bd3f6a0" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3" is_pre_election: true
I20240426 09:53:40.550442 9041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8a55ea6e5429093a5ec516bd3f6a0" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d" is_pre_election: true
I20240426 09:53:40.551234 9041 raft_consensus.cc:2379] T 4af8a55ea6e5429093a5ec516bd3f6a0 P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cdbe92f5959a47dda817bbec2148eab9 in current term 1: Already voted for candidate 9e98a77341c74d6d90d01cd3e39586b3 in this term.
I20240426 09:53:40.552371 8859 leader_election.cc:302] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cdbe92f5959a47dda817bbec2148eab9; no voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d
I20240426 09:53:40.553169 9134 raft_consensus.cc:3037] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.560853 9134 raft_consensus.cc:2726] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:53:40.581677 9041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec"
dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
mode: GRACEFUL
new_leader_uuid: "cdbe92f5959a47dda817bbec2148eab9"
from {username='slave'} at 127.0.0.1:37868
I20240426 09:53:40.582168 9041 raft_consensus.cc:595] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 1 LEADER]: Received request to transfer leadership to TS cdbe92f5959a47dda817bbec2148eab9
I20240426 09:53:40.596925 8968 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73"
dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
mode: GRACEFUL
new_leader_uuid: "cdbe92f5959a47dda817bbec2148eab9"
from {username='slave'} at 127.0.0.1:44598
I20240426 09:53:40.597400 8968 raft_consensus.cc:595] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 LEADER]: Received request to transfer leadership to TS cdbe92f5959a47dda817bbec2148eab9
I20240426 09:53:40.602128 8968 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009"
dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
mode: GRACEFUL
new_leader_uuid: "cdbe92f5959a47dda817bbec2148eab9"
from {username='slave'} at 127.0.0.1:44598
I20240426 09:53:40.602532 8968 raft_consensus.cc:595] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 LEADER]: Received request to transfer leadership to TS cdbe92f5959a47dda817bbec2148eab9
I20240426 09:53:40.616530 8892 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "431d4df7c67747fcb0812c91964c801b"
dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
mode: GRACEFUL
new_leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
from {username='slave'} at 127.0.0.1:53952
I20240426 09:53:40.617110 8892 raft_consensus.cc:595] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 1 LEADER]: Received request to transfer leadership to TS 9e98a77341c74d6d90d01cd3e39586b3
I20240426 09:53:40.622310 8892 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df73379416114f8ab7efd3caf9c22e7e"
dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
mode: GRACEFUL
new_leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
from {username='slave'} at 127.0.0.1:53952
I20240426 09:53:40.622890 8892 raft_consensus.cc:595] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 1 LEADER]: Received request to transfer leadership to TS 9e98a77341c74d6d90d01cd3e39586b3
I20240426 09:53:40.628237 9041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "97deaa8654f64ad2a63314ffaa91d955"
dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
mode: GRACEFUL
new_leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
from {username='slave'} at 127.0.0.1:37868
I20240426 09:53:40.628798 9041 raft_consensus.cc:595] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 1 LEADER]: Received request to transfer leadership to TS 9e98a77341c74d6d90d01cd3e39586b3
I20240426 09:53:40.630455 9148 raft_consensus.cc:482] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:53:40.630956 9148 raft_consensus.cc:504] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.633518 9148 leader_election.cc:288] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991)
I20240426 09:53:40.634925 8892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431d4df7c67747fcb0812c91964c801b" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbe92f5959a47dda817bbec2148eab9" is_pre_election: true
I20240426 09:53:40.635183 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431d4df7c67747fcb0812c91964c801b" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3" is_pre_election: true
I20240426 09:53:40.635855 8968 raft_consensus.cc:2379] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d99df4d00f1e42699f4e40da65bbc33d in current term 1: Already voted for candidate cdbe92f5959a47dda817bbec2148eab9 in this term.
I20240426 09:53:40.637532 9007 leader_election.cc:302] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d99df4d00f1e42699f4e40da65bbc33d; no voters: 9e98a77341c74d6d90d01cd3e39586b3, cdbe92f5959a47dda817bbec2148eab9
I20240426 09:53:40.638919 9148 raft_consensus.cc:3037] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.640535 9087 consensus_queue.cc:1035] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:40.640830 9124 consensus_queue.cc:1035] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 09:53:40.646875 9148 raft_consensus.cc:2726] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:53:40.653560 8892 raft_consensus.cc:3037] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.656141 9124 raft_consensus.cc:979] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3: : Instructing follower cdbe92f5959a47dda817bbec2148eab9 to start an election
I20240426 09:53:40.656538 9124 raft_consensus.cc:1067] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 LEADER]: Signalling peer cdbe92f5959a47dda817bbec2148eab9 to start an election
I20240426 09:53:40.664252 8893 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73"
dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
from {username='slave'} at 127.0.0.1:53924
I20240426 09:53:40.664808 8893 raft_consensus.cc:482] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:40.665153 8893 raft_consensus.cc:3037] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:40.670426 9124 consensus_queue.cc:1035] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:40.670632 9110 consensus_queue.cc:1035] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:40.674518 8893 raft_consensus.cc:504] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.677273 8892 raft_consensus.cc:1226] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Rejecting Update request from peer 9e98a77341c74d6d90d01cd3e39586b3 for earlier term 1. Current term is 2. Ops: []
I20240426 09:53:40.677860 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:53:40.678421 8968 raft_consensus.cc:3032] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:53:40.679109 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:40.679647 9042 raft_consensus.cc:3037] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:40.680255 8893 leader_election.cc:288] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 2 election: Requested vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.678723 8968 raft_consensus.cc:729] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:40.683172 8968 consensus_queue.cc:260] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.686908 9124 consensus_queue.cc:1046] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:40.689006 9087 consensus_queue.cc:1035] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:40.690955 8968 raft_consensus.cc:3037] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:40.698395 9087 consensus_queue.cc:1035] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:40.703637 9042 raft_consensus.cc:2454] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 2.
I20240426 09:53:40.704375 9041 raft_consensus.cc:1226] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Rejecting Update request from peer 9e98a77341c74d6d90d01cd3e39586b3 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20240426 09:53:40.710243 8968 raft_consensus.cc:2454] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 2.
I20240426 09:53:40.712574 8859 leader_election.cc:302] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdbe92f5959a47dda817bbec2148eab9, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:40.713342 9134 raft_consensus.cc:2781] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:53:40.713773 9134 raft_consensus.cc:686] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 2 LEADER]: Becoming Leader. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Running, Role: LEADER
I20240426 09:53:40.714558 9134 consensus_queue.cc:237] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.717633 9148 consensus_queue.cc:1035] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:40.738072 8790 catalog_manager.cc:5484] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 reported cstate change: term changed from 1 to 2, leader changed from 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130) to cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129). New cstate: current_term: 2 leader_uuid: "cdbe92f5959a47dda817bbec2148eab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:40.740775 9134 raft_consensus.cc:482] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:53:40.741608 9134 raft_consensus.cc:504] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.741129 9124 raft_consensus.cc:979] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3: : Instructing follower cdbe92f5959a47dda817bbec2148eab9 to start an election
I20240426 09:53:40.743407 9134 leader_election.cc:288] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.743710 9124 raft_consensus.cc:1067] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 LEADER]: Signalling peer cdbe92f5959a47dda817bbec2148eab9 to start an election
I20240426 09:53:40.746897 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97deaa8654f64ad2a63314ffaa91d955" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d" is_pre_election: true
I20240426 09:53:40.749228 8892 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009"
dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
from {username='slave'} at 127.0.0.1:53924
I20240426 09:53:40.751051 9134 consensus_queue.cc:1035] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:40.752388 8892 raft_consensus.cc:482] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:40.752763 8892 raft_consensus.cc:3037] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:40.755874 9095 raft_consensus.cc:979] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d: : Instructing follower 9e98a77341c74d6d90d01cd3e39586b3 to start an election
I20240426 09:53:40.756335 9095 raft_consensus.cc:1067] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 1 LEADER]: Signalling peer 9e98a77341c74d6d90d01cd3e39586b3 to start an election
I20240426 09:53:40.764863 8966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97deaa8654f64ad2a63314ffaa91d955" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3" is_pre_election: true
I20240426 09:53:40.766346 8858 leader_election.cc:302] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cdbe92f5959a47dda817bbec2148eab9; no voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d
I20240426 09:53:40.768452 9136 raft_consensus.cc:2726] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:53:40.769414 8891 raft_consensus.cc:3037] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:53:40.770659 8892 raft_consensus.cc:504] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.775002 8966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:53:40.775547 8966 raft_consensus.cc:3032] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:53:40.775848 8966 raft_consensus.cc:729] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:40.776554 8966 consensus_queue.cc:260] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.781419 8967 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97deaa8654f64ad2a63314ffaa91d955"
dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
from {username='slave'} at 127.0.0.1:44576
I20240426 09:53:40.781962 8967 raft_consensus.cc:482] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:40.782317 8967 raft_consensus.cc:3037] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:40.790796 9112 consensus_queue.cc:1035] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:40.777639 8966 raft_consensus.cc:3037] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:40.793624 8967 raft_consensus.cc:504] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.796439 8893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97deaa8654f64ad2a63314ffaa91d955" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
I20240426 09:53:40.796624 9040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97deaa8654f64ad2a63314ffaa91d955" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:40.797176 9040 raft_consensus.cc:3032] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:53:40.797485 9040 raft_consensus.cc:729] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Running, Role: LEADER
I20240426 09:53:40.798199 9040 consensus_queue.cc:260] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.799264 9040 raft_consensus.cc:3037] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:40.800120 8967 leader_election.cc:288] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 2 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.808557 9136 consensus_queue.cc:1035] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:40.808844 9040 raft_consensus.cc:2454] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 2.
I20240426 09:53:40.810030 8934 leader_election.cc:302] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters: cdbe92f5959a47dda817bbec2148eab9
I20240426 09:53:40.810806 9087 raft_consensus.cc:2781] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:53:40.811014 8892 leader_election.cc:288] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 2 election: Requested vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.811328 9087 raft_consensus.cc:686] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 LEADER]: Becoming Leader. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:40.811753 9041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:40.812348 9041 raft_consensus.cc:3037] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:40.812106 9087 consensus_queue.cc:237] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.816051 8966 raft_consensus.cc:2454] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 2.
I20240426 09:53:40.820282 8858 leader_election.cc:302] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, cdbe92f5959a47dda817bbec2148eab9; no voters:
I20240426 09:53:40.821285 8787 catalog_manager.cc:5484] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 reported cstate change: term changed from 1 to 2, leader changed from d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131) to 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130). New cstate: current_term: 2 leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:40.825636 9041 raft_consensus.cc:2454] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 2.
I20240426 09:53:40.829435 9136 raft_consensus.cc:2781] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:53:40.829849 9136 raft_consensus.cc:686] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 2 LEADER]: Becoming Leader. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Running, Role: LEADER
I20240426 09:53:40.830489 9136 consensus_queue.cc:237] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.834265 9136 consensus_queue.cc:1035] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:40.844173 8788 catalog_manager.cc:5484] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 reported cstate change: term changed from 1 to 2, leader changed from 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130) to cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129). New cstate: current_term: 2 leader_uuid: "cdbe92f5959a47dda817bbec2148eab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:40.851934 9136 consensus_queue.cc:1035] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:40.860353 9112 consensus_queue.cc:1035] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:40.863735 9094 raft_consensus.cc:979] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9: : Instructing follower 9e98a77341c74d6d90d01cd3e39586b3 to start an election
I20240426 09:53:40.867245 9134 raft_consensus.cc:979] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9: : Instructing follower 9e98a77341c74d6d90d01cd3e39586b3 to start an election
I20240426 09:53:40.867669 9134 raft_consensus.cc:1067] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 1 LEADER]: Signalling peer 9e98a77341c74d6d90d01cd3e39586b3 to start an election
I20240426 09:53:40.868330 9134 raft_consensus.cc:1067] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 1 LEADER]: Signalling peer 9e98a77341c74d6d90d01cd3e39586b3 to start an election
I20240426 09:53:40.878592 9165 consensus_queue.cc:1035] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:40.889406 8968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df73379416114f8ab7efd3caf9c22e7e"
dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
from {username='slave'} at 127.0.0.1:44588
I20240426 09:53:40.889946 8968 raft_consensus.cc:482] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:40.890308 8968 raft_consensus.cc:3037] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:40.895278 8966 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "431d4df7c67747fcb0812c91964c801b"
dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
from {username='slave'} at 127.0.0.1:44588
I20240426 09:53:40.895812 8966 raft_consensus.cc:482] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:40.896188 8966 raft_consensus.cc:3037] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:40.902946 9165 consensus_queue.cc:1035] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 09:53:40.906116 9112 consensus_queue.cc:1035] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:40.902391 8966 raft_consensus.cc:504] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.909157 8891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431d4df7c67747fcb0812c91964c801b" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
I20240426 09:53:40.909601 8891 raft_consensus.cc:3032] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:53:40.909891 8891 raft_consensus.cc:729] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Running, Role: LEADER
I20240426 09:53:40.910512 8891 consensus_queue.cc:260] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.911592 8891 raft_consensus.cc:3037] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:40.919833 9112 raft_consensus.cc:979] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d: : Instructing follower cdbe92f5959a47dda817bbec2148eab9 to start an election
I20240426 09:53:40.920243 9112 raft_consensus.cc:1067] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 1 LEADER]: Signalling peer cdbe92f5959a47dda817bbec2148eab9 to start an election
I20240426 09:53:40.920218 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431d4df7c67747fcb0812c91964c801b" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:40.920914 9042 raft_consensus.cc:3037] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:40.925850 8891 raft_consensus.cc:2454] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 2.
I20240426 09:53:40.926680 8933 leader_election.cc:302] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, cdbe92f5959a47dda817bbec2148eab9; no voters:
I20240426 09:53:40.927297 9042 raft_consensus.cc:2454] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 2.
I20240426 09:53:40.928326 9087 raft_consensus.cc:2781] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:53:40.928680 9087 raft_consensus.cc:686] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 LEADER]: Becoming Leader. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:40.928606 8892 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec"
dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
from {username='slave'} at 127.0.0.1:53940
I20240426 09:53:40.929227 8892 raft_consensus.cc:482] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:40.929589 8892 raft_consensus.cc:3037] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:40.929330 9087 consensus_queue.cc:237] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.933001 8966 leader_election.cc:288] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 2 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.937254 8892 raft_consensus.cc:504] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.917857 8968 raft_consensus.cc:504] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.939747 8966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:53:40.940305 8966 raft_consensus.cc:3037] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:40.942162 8891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df73379416114f8ab7efd3caf9c22e7e" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
I20240426 09:53:40.942785 8891 raft_consensus.cc:3032] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:53:40.942868 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df73379416114f8ab7efd3caf9c22e7e" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:40.943181 8891 raft_consensus.cc:729] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Running, Role: LEADER
I20240426 09:53:40.943456 9042 raft_consensus.cc:3037] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:40.944134 8891 consensus_queue.cc:260] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.945259 8891 raft_consensus.cc:3037] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:40.944720 8790 catalog_manager.cc:5484] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 reported cstate change: term changed from 1 to 2, leader changed from cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129) to 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130). New cstate: current_term: 2 leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:40.949811 9042 raft_consensus.cc:2454] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 2.
I20240426 09:53:40.950810 8934 leader_election.cc:302] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:40.951534 9087 raft_consensus.cc:2781] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:53:40.951990 9087 raft_consensus.cc:686] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 LEADER]: Becoming Leader. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:40.952785 9087 consensus_queue.cc:237] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.955363 8966 raft_consensus.cc:2454] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 2.
I20240426 09:53:40.956331 8858 leader_election.cc:302] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, cdbe92f5959a47dda817bbec2148eab9; no voters:
I20240426 09:53:40.957142 9165 raft_consensus.cc:2781] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:53:40.957809 9165 raft_consensus.cc:686] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 2 LEADER]: Becoming Leader. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Running, Role: LEADER
I20240426 09:53:40.958420 8891 raft_consensus.cc:2454] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 2.
I20240426 09:53:40.958627 9165 consensus_queue.cc:237] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.959904 8968 leader_election.cc:288] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 2 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.961275 8892 leader_election.cc:288] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 2 election: Requested vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:40.962193 9041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:40.962730 9041 raft_consensus.cc:3032] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:53:40.963055 9041 raft_consensus.cc:729] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Running, Role: LEADER
I20240426 09:53:40.963753 9041 consensus_queue.cc:260] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:40.964919 9041 raft_consensus.cc:3037] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:40.964450 8790 catalog_manager.cc:5484] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 reported cstate change: term changed from 1 to 2, leader changed from cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129) to 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130). New cstate: current_term: 2 leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:40.969938 8788 catalog_manager.cc:5484] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 reported cstate change: term changed from 1 to 2, leader changed from d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131) to cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129). New cstate: current_term: 2 leader_uuid: "cdbe92f5959a47dda817bbec2148eab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:40.970705 9041 raft_consensus.cc:2454] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 2.
I20240426 09:53:41.152827 8968 raft_consensus.cc:1261] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Refusing update from remote peer cdbe92f5959a47dda817bbec2148eab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:53:41.153946 9165 consensus_queue.cc:1035] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:53:41.162036 9041 raft_consensus.cc:1261] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Refusing update from remote peer cdbe92f5959a47dda817bbec2148eab9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:53:41.164557 9136 consensus_queue.cc:1035] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:53:41.243139 8892 raft_consensus.cc:3037] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:41.249926 8892 raft_consensus.cc:1261] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Refusing update from remote peer 9e98a77341c74d6d90d01cd3e39586b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:53:41.251168 9110 consensus_queue.cc:1035] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:53:41.260772 9042 raft_consensus.cc:1261] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Refusing update from remote peer 9e98a77341c74d6d90d01cd3e39586b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:53:41.262054 9124 consensus_queue.cc:1035] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:53:41.319653 9042 raft_consensus.cc:1261] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Refusing update from remote peer cdbe92f5959a47dda817bbec2148eab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:53:41.320945 9165 consensus_queue.cc:1035] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:53:41.331552 8968 raft_consensus.cc:1261] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Refusing update from remote peer cdbe92f5959a47dda817bbec2148eab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:53:41.335063 9165 consensus_queue.cc:1035] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:53:41.357913 8968 raft_consensus.cc:1261] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Refusing update from remote peer cdbe92f5959a47dda817bbec2148eab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:53:41.358928 9165 consensus_queue.cc:1035] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:53:41.365259 9042 raft_consensus.cc:1261] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Refusing update from remote peer cdbe92f5959a47dda817bbec2148eab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:53:41.366909 9136 consensus_queue.cc:1035] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:53:41.383996 9042 raft_consensus.cc:1261] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Refusing update from remote peer 9e98a77341c74d6d90d01cd3e39586b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:53:41.384989 9124 consensus_queue.cc:1035] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:53:41.391252 8892 raft_consensus.cc:1261] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Refusing update from remote peer 9e98a77341c74d6d90d01cd3e39586b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:53:41.392549 9087 consensus_queue.cc:1035] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:53:41.400202 9042 raft_consensus.cc:1261] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Refusing update from remote peer 9e98a77341c74d6d90d01cd3e39586b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:53:41.401813 9087 consensus_queue.cc:1035] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:53:41.408987 8892 raft_consensus.cc:1261] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Refusing update from remote peer 9e98a77341c74d6d90d01cd3e39586b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:53:41.410168 9087 consensus_queue.cc:1035] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:53:43.652336 8758 auto_leader_rebalancer-test.cc:261] The leader distribution is
I20240426 09:53:43.653298 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:53:43.671463 8892 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec"
dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
mode: GRACEFUL
new_leader_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
from {username='slave'} at 127.0.0.1:53956
I20240426 09:53:43.672052 8892 raft_consensus.cc:595] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 2 LEADER]: Received request to transfer leadership to TS d99df4d00f1e42699f4e40da65bbc33d
I20240426 09:53:43.680364 8968 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df73379416114f8ab7efd3caf9c22e7e"
dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
mode: GRACEFUL
new_leader_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
from {username='slave'} at 127.0.0.1:44602
I20240426 09:53:43.680773 8968 raft_consensus.cc:595] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 LEADER]: Received request to transfer leadership to TS d99df4d00f1e42699f4e40da65bbc33d
I20240426 09:53:43.681589 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240426 09:53:43.682014 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:53:43.935966 9094 raft_consensus.cc:979] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9: : Instructing follower d99df4d00f1e42699f4e40da65bbc33d to start an election
I20240426 09:53:43.936349 9174 raft_consensus.cc:1067] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 2 LEADER]: Signalling peer d99df4d00f1e42699f4e40da65bbc33d to start an election
I20240426 09:53:43.937686 9042 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec"
dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
from {username='slave'} at 127.0.0.1:37854
I20240426 09:53:43.938115 9042 raft_consensus.cc:482] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:43.938390 9042 raft_consensus.cc:3037] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:43.942409 9042 raft_consensus.cc:504] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:43.943868 9042 leader_election.cc:288] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 3 election: Requested vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507)
I20240426 09:53:43.944463 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:53:43.944875 8892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
I20240426 09:53:43.945037 8968 raft_consensus.cc:3037] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:43.945365 8892 raft_consensus.cc:3032] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:53:43.945637 8892 raft_consensus.cc:729] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Running, Role: LEADER
I20240426 09:53:43.946170 8892 consensus_queue.cc:260] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:43.947011 8892 raft_consensus.cc:3037] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:43.949205 8968 raft_consensus.cc:2454] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 3.
I20240426 09:53:43.950151 9007 leader_election.cc:302] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:43.950930 9112 raft_consensus.cc:2781] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:53:43.951283 8892 raft_consensus.cc:2454] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 3.
I20240426 09:53:43.951995 9112 raft_consensus.cc:686] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 3 LEADER]: Becoming Leader. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Running, Role: LEADER
I20240426 09:53:43.952693 9112 consensus_queue.cc:237] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:43.958827 8788 catalog_manager.cc:5484] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d reported cstate change: term changed from 2 to 3, leader changed from cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129) to d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131). New cstate: current_term: 3 leader_uuid: "d99df4d00f1e42699f4e40da65bbc33d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: HEALTHY } } }
I20240426 09:53:44.189656 9181 raft_consensus.cc:979] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3: : Instructing follower d99df4d00f1e42699f4e40da65bbc33d to start an election
I20240426 09:53:44.190042 9170 raft_consensus.cc:1067] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 LEADER]: Signalling peer d99df4d00f1e42699f4e40da65bbc33d to start an election
I20240426 09:53:44.191181 9042 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df73379416114f8ab7efd3caf9c22e7e"
dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
from {username='slave'} at 127.0.0.1:37844
I20240426 09:53:44.191612 9042 raft_consensus.cc:482] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:44.191885 9042 raft_consensus.cc:3037] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:44.195915 9042 raft_consensus.cc:504] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:44.197206 9042 leader_election.cc:288] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 3 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991)
I20240426 09:53:44.197947 8892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df73379416114f8ab7efd3caf9c22e7e" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
I20240426 09:53:44.198074 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df73379416114f8ab7efd3caf9c22e7e" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:53:44.198382 8892 raft_consensus.cc:3037] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:44.198575 8968 raft_consensus.cc:3032] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:53:44.198940 8968 raft_consensus.cc:729] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:44.199493 8968 consensus_queue.cc:260] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:44.200289 8968 raft_consensus.cc:3037] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:44.202695 8892 raft_consensus.cc:2454] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 3.
I20240426 09:53:44.203529 9007 leader_election.cc:302] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdbe92f5959a47dda817bbec2148eab9, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:44.204241 9112 raft_consensus.cc:2781] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:53:44.204592 9112 raft_consensus.cc:686] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 3 LEADER]: Becoming Leader. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Running, Role: LEADER
I20240426 09:53:44.204787 8968 raft_consensus.cc:2454] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 3.
I20240426 09:53:44.205271 9112 consensus_queue.cc:237] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:44.213478 8788 catalog_manager.cc:5484] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d reported cstate change: term changed from 2 to 3, leader changed from 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130) to d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131). New cstate: current_term: 3 leader_uuid: "d99df4d00f1e42699f4e40da65bbc33d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: HEALTHY } } }
I20240426 09:53:44.459744 8892 raft_consensus.cc:1261] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Refusing update from remote peer d99df4d00f1e42699f4e40da65bbc33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:53:44.460738 9112 consensus_queue.cc:1035] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:53:44.470499 8968 raft_consensus.cc:1261] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Refusing update from remote peer d99df4d00f1e42699f4e40da65bbc33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:53:44.471943 9186 consensus_queue.cc:1035] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:53:44.682693 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:53:44.687511 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:53:44.687938 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:53:44.767521 8892 raft_consensus.cc:1261] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Refusing update from remote peer d99df4d00f1e42699f4e40da65bbc33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:53:44.768605 9112 consensus_queue.cc:1035] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:53:44.778101 8968 raft_consensus.cc:1261] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Refusing update from remote peer d99df4d00f1e42699f4e40da65bbc33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:53:44.779392 9112 consensus_queue.cc:1035] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:53:45.688616 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:53:45.693202 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:53:45.693552 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:53:46.694197 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:53:46.699242 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:53:46.699599 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:53:47.700515 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:53:47.706043 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:53:47.706430 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:53:48.707230 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:53:48.714520 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:53:48.715032 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:53:49.715966 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:53:49.724126 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:53:49.724639 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:53:50.725831 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:53:50.732719 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:53:50.733222 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:53:51.733966 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:53:51.738899 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:53:51.739372 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:53:52.740128 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:53:52.744982 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:53:52.745306 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:53:53.765372 8758 auto_leader_rebalancer-test.cc:280] The leader distribution is
I20240426 09:53:53.769742 9042 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec"
dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
mode: GRACEFUL
new_leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
from {username='slave'} at 127.0.0.1:37868
I20240426 09:53:53.770488 9042 raft_consensus.cc:595] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 3 LEADER]: Received request to transfer leadership to TS 9e98a77341c74d6d90d01cd3e39586b3
I20240426 09:53:53.774447 8891 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91ee2df31bb64b6a9536f91149a87c0d"
dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
mode: GRACEFUL
new_leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
from {username='slave'} at 127.0.0.1:53952
I20240426 09:53:53.774900 8891 raft_consensus.cc:595] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 1 LEADER]: Received request to transfer leadership to TS 9e98a77341c74d6d90d01cd3e39586b3
I20240426 09:53:53.778543 8891 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73"
dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
mode: GRACEFUL
new_leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
from {username='slave'} at 127.0.0.1:53952
I20240426 09:53:53.778963 8891 raft_consensus.cc:595] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 2 LEADER]: Received request to transfer leadership to TS 9e98a77341c74d6d90d01cd3e39586b3
I20240426 09:53:53.782352 8891 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009"
dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
mode: GRACEFUL
new_leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
from {username='slave'} at 127.0.0.1:53952
I20240426 09:53:53.782735 8891 raft_consensus.cc:595] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 2 LEADER]: Received request to transfer leadership to TS 9e98a77341c74d6d90d01cd3e39586b3
I20240426 09:53:53.790127 9042 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df73379416114f8ab7efd3caf9c22e7e"
dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
mode: GRACEFUL
new_leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
from {username='slave'} at 127.0.0.1:37868
I20240426 09:53:53.790518 9042 raft_consensus.cc:595] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 3 LEADER]: Received request to transfer leadership to TS 9e98a77341c74d6d90d01cd3e39586b3
I20240426 09:53:53.820230 9194 raft_consensus.cc:979] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d: : Instructing follower 9e98a77341c74d6d90d01cd3e39586b3 to start an election
I20240426 09:53:53.820726 9175 raft_consensus.cc:1067] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 3 LEADER]: Signalling peer 9e98a77341c74d6d90d01cd3e39586b3 to start an election
I20240426 09:53:53.821919 8968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df73379416114f8ab7efd3caf9c22e7e"
dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
from {username='slave'} at 127.0.0.1:44576
I20240426 09:53:53.822376 8968 raft_consensus.cc:482] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:53.822679 8968 raft_consensus.cc:3037] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:53:53.827090 8968 raft_consensus.cc:504] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:53.828727 8968 leader_election.cc:288] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 4 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:53.829715 8891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df73379416114f8ab7efd3caf9c22e7e" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
I20240426 09:53:53.829706 9041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df73379416114f8ab7efd3caf9c22e7e" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:53.830688 9041 raft_consensus.cc:3032] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:53:53.830976 9041 raft_consensus.cc:729] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Running, Role: LEADER
I20240426 09:53:53.831508 8891 raft_consensus.cc:3037] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:53:53.831561 9041 consensus_queue.cc:260] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:53.832505 9041 raft_consensus.cc:3037] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:53:53.836038 8891 raft_consensus.cc:2454] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 4.
I20240426 09:53:53.837110 9175 raft_consensus.cc:979] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d: : Instructing follower 9e98a77341c74d6d90d01cd3e39586b3 to start an election
I20240426 09:53:53.837231 8933 leader_election.cc:302] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, cdbe92f5959a47dda817bbec2148eab9; no voters:
I20240426 09:53:53.837451 9041 raft_consensus.cc:2454] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 4.
I20240426 09:53:53.837635 9194 raft_consensus.cc:1067] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 3 LEADER]: Signalling peer 9e98a77341c74d6d90d01cd3e39586b3 to start an election
I20240426 09:53:53.838176 9170 raft_consensus.cc:2781] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:53:53.839108 8968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec"
dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
from {username='slave'} at 127.0.0.1:44576
I20240426 09:53:53.839205 9170 raft_consensus.cc:686] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 LEADER]: Becoming Leader. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:53.839517 8968 raft_consensus.cc:482] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:53.839876 8968 raft_consensus.cc:3037] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:53:53.839985 9170 consensus_queue.cc:237] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:53.844026 8968 raft_consensus.cc:504] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:53.845520 8968 leader_election.cc:288] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 4 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:53.846174 8891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
I20240426 09:53:53.846526 9041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:53.846801 8891 raft_consensus.cc:3037] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:53:53.847034 9041 raft_consensus.cc:3032] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:53:53.847393 9041 raft_consensus.cc:729] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Running, Role: LEADER
I20240426 09:53:53.847087 8788 catalog_manager.cc:5484] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 reported cstate change: term changed from 3 to 4, leader changed from d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131) to 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130). New cstate: current_term: 4 leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:53.847978 9041 consensus_queue.cc:260] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:53.848877 9041 raft_consensus.cc:3037] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:53:53.852113 8891 raft_consensus.cc:2454] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 4.
I20240426 09:53:53.852945 8933 leader_election.cc:302] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, cdbe92f5959a47dda817bbec2148eab9; no voters:
I20240426 09:53:53.853585 9170 raft_consensus.cc:2781] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:53:53.853917 9170 raft_consensus.cc:686] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 LEADER]: Becoming Leader. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:53.854223 9041 raft_consensus.cc:2454] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 4.
I20240426 09:53:53.854591 9170 consensus_queue.cc:237] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:53.862668 9202 raft_consensus.cc:979] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9: : Instructing follower 9e98a77341c74d6d90d01cd3e39586b3 to start an election
I20240426 09:53:53.863133 9174 raft_consensus.cc:1067] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 2 LEADER]: Signalling peer 9e98a77341c74d6d90d01cd3e39586b3 to start an election
I20240426 09:53:53.864292 8968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009"
dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
from {username='slave'} at 127.0.0.1:44588
I20240426 09:53:53.864651 8968 raft_consensus.cc:482] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:53.864893 8968 raft_consensus.cc:3037] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:53.864764 8788 catalog_manager.cc:5484] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 reported cstate change: term changed from 3 to 4, leader changed from d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131) to 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130). New cstate: current_term: 4 leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:53.868435 8968 raft_consensus.cc:504] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:53.869758 8968 leader_election.cc:288] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 3 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:53.871001 9041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:53.871012 8891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
I20240426 09:53:53.871702 8891 raft_consensus.cc:3032] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:53:53.871659 9041 raft_consensus.cc:3037] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:53.872102 8891 raft_consensus.cc:729] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Running, Role: LEADER
I20240426 09:53:53.872826 8891 consensus_queue.cc:260] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:53.874047 8891 raft_consensus.cc:3037] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:53.877668 9041 raft_consensus.cc:2454] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 3.
I20240426 09:53:53.878636 8934 leader_election.cc:302] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:53.879365 9170 raft_consensus.cc:2781] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:53:53.879673 9170 raft_consensus.cc:686] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 LEADER]: Becoming Leader. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:53.879745 8891 raft_consensus.cc:2454] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 3.
I20240426 09:53:53.880306 9170 consensus_queue.cc:237] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:53.885691 8788 catalog_manager.cc:5484] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 reported cstate change: term changed from 2 to 3, leader changed from cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129) to 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130). New cstate: current_term: 3 leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:53.989540 9202 raft_consensus.cc:979] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9: : Instructing follower 9e98a77341c74d6d90d01cd3e39586b3 to start an election
I20240426 09:53:53.989971 9174 raft_consensus.cc:1067] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 1 LEADER]: Signalling peer 9e98a77341c74d6d90d01cd3e39586b3 to start an election
I20240426 09:53:53.991101 8968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "91ee2df31bb64b6a9536f91149a87c0d"
dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
from {username='slave'} at 127.0.0.1:44588
I20240426 09:53:53.991487 8968 raft_consensus.cc:482] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:53.991719 8968 raft_consensus.cc:3037] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:53.995339 8968 raft_consensus.cc:504] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:53.996534 8968 leader_election.cc:288] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 2 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:53.997112 8891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ee2df31bb64b6a9536f91149a87c0d" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
I20240426 09:53:53.997506 9041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ee2df31bb64b6a9536f91149a87c0d" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:53.997642 8891 raft_consensus.cc:3032] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:53:53.997910 8891 raft_consensus.cc:729] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Running, Role: LEADER
I20240426 09:53:53.997956 9041 raft_consensus.cc:3037] T 91ee2df31bb64b6a9536f91149a87c0d P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:53.998509 8891 consensus_queue.cc:260] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:53.999434 8891 raft_consensus.cc:3037] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:53:54.002008 9041 raft_consensus.cc:2454] T 91ee2df31bb64b6a9536f91149a87c0d P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 2.
I20240426 09:53:54.002839 8934 leader_election.cc:302] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:54.003566 9170 raft_consensus.cc:2781] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:53:54.003744 8891 raft_consensus.cc:2454] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 2.
I20240426 09:53:54.003887 9170 raft_consensus.cc:686] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 LEADER]: Becoming Leader. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:54.004750 9170 consensus_queue.cc:237] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:54.010311 8788 catalog_manager.cc:5484] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 reported cstate change: term changed from 1 to 2, leader changed from cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129) to 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130). New cstate: current_term: 2 leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:54.176813 9202 raft_consensus.cc:979] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9: : Instructing follower 9e98a77341c74d6d90d01cd3e39586b3 to start an election
I20240426 09:53:54.177220 9174 raft_consensus.cc:1067] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 2 LEADER]: Signalling peer 9e98a77341c74d6d90d01cd3e39586b3 to start an election
I20240426 09:53:54.178452 8968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73"
dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
from {username='slave'} at 127.0.0.1:44588
I20240426 09:53:54.178860 8968 raft_consensus.cc:482] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:54.179126 8968 raft_consensus.cc:3037] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:54.182708 8968 raft_consensus.cc:504] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:54.183913 8968 leader_election.cc:288] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 3 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:54.184731 9041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:54.184803 8891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
I20240426 09:53:54.185204 9041 raft_consensus.cc:3037] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:54.185256 8891 raft_consensus.cc:3032] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:53:54.185647 8891 raft_consensus.cc:729] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Running, Role: LEADER
I20240426 09:53:54.186383 8891 consensus_queue.cc:260] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:54.187323 8891 raft_consensus.cc:3037] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:54.189857 9041 raft_consensus.cc:2454] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 3.
I20240426 09:53:54.190706 8934 leader_election.cc:302] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:54.191344 9170 raft_consensus.cc:2781] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:53:54.191795 9170 raft_consensus.cc:686] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 LEADER]: Becoming Leader. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:54.192201 8891 raft_consensus.cc:2454] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 3.
I20240426 09:53:54.192489 9170 consensus_queue.cc:237] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:54.197888 8788 catalog_manager.cc:5484] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 reported cstate change: term changed from 2 to 3, leader changed from cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129) to 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130). New cstate: current_term: 3 leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:54.307684 8891 raft_consensus.cc:1261] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 4 FOLLOWER]: Refusing update from remote peer 9e98a77341c74d6d90d01cd3e39586b3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:53:54.308614 9170 consensus_queue.cc:1035] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:53:54.313890 8892 raft_consensus.cc:1261] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 4 FOLLOWER]: Refusing update from remote peer 9e98a77341c74d6d90d01cd3e39586b3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:53:54.317673 9170 consensus_queue.cc:1035] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:53:54.331236 9041 raft_consensus.cc:1261] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Refusing update from remote peer 9e98a77341c74d6d90d01cd3e39586b3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:53:54.332690 9170 consensus_queue.cc:1035] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:53:54.334681 9041 raft_consensus.cc:1261] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Refusing update from remote peer 9e98a77341c74d6d90d01cd3e39586b3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:53:54.340150 9211 consensus_queue.cc:1035] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:53:54.402962 8891 raft_consensus.cc:1261] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Refusing update from remote peer 9e98a77341c74d6d90d01cd3e39586b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:53:54.404042 9211 consensus_queue.cc:1035] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:53:54.406284 8892 raft_consensus.cc:1261] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Refusing update from remote peer 9e98a77341c74d6d90d01cd3e39586b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:53:54.407217 9170 consensus_queue.cc:1035] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:53:54.411800 9041 raft_consensus.cc:1261] T 91ee2df31bb64b6a9536f91149a87c0d P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Refusing update from remote peer 9e98a77341c74d6d90d01cd3e39586b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:53:54.413545 9211 consensus_queue.cc:1035] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:53:54.424800 9042 raft_consensus.cc:1261] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Refusing update from remote peer 9e98a77341c74d6d90d01cd3e39586b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:53:54.426540 9208 consensus_queue.cc:1035] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:53:54.655257 9042 raft_consensus.cc:1261] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Refusing update from remote peer 9e98a77341c74d6d90d01cd3e39586b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:53:54.656145 9208 consensus_queue.cc:1035] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:53:54.663607 8892 raft_consensus.cc:1261] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Refusing update from remote peer 9e98a77341c74d6d90d01cd3e39586b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:53:54.664867 9208 consensus_queue.cc:1035] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:53:56.833417 8758 auto_leader_rebalancer-test.cc:295] The leader distribution is
I20240426 09:53:56.833979 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:53:56.839834 8968 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73"
dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
mode: GRACEFUL
new_leader_uuid: "cdbe92f5959a47dda817bbec2148eab9"
from {username='slave'} at 127.0.0.1:44602
I20240426 09:53:56.840229 8968 raft_consensus.cc:595] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 LEADER]: Received request to transfer leadership to TS cdbe92f5959a47dda817bbec2148eab9
I20240426 09:53:56.842084 8968 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec"
dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
mode: GRACEFUL
new_leader_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
from {username='slave'} at 127.0.0.1:44602
I20240426 09:53:56.842476 8968 raft_consensus.cc:595] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 LEADER]: Received request to transfer leadership to TS d99df4d00f1e42699f4e40da65bbc33d
I20240426 09:53:56.844156 8968 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "431d4df7c67747fcb0812c91964c801b"
dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
mode: GRACEFUL
new_leader_uuid: "cdbe92f5959a47dda817bbec2148eab9"
from {username='slave'} at 127.0.0.1:44602
I20240426 09:53:56.844519 8968 raft_consensus.cc:595] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 LEADER]: Received request to transfer leadership to TS cdbe92f5959a47dda817bbec2148eab9
I20240426 09:53:56.846146 8968 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009"
dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
mode: GRACEFUL
new_leader_uuid: "cdbe92f5959a47dda817bbec2148eab9"
from {username='slave'} at 127.0.0.1:44602
I20240426 09:53:56.846494 8968 raft_consensus.cc:595] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 LEADER]: Received request to transfer leadership to TS cdbe92f5959a47dda817bbec2148eab9
I20240426 09:53:56.848111 8968 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "97deaa8654f64ad2a63314ffaa91d955"
dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
mode: GRACEFUL
new_leader_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
from {username='slave'} at 127.0.0.1:44602
I20240426 09:53:56.848459 8968 raft_consensus.cc:595] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 LEADER]: Received request to transfer leadership to TS d99df4d00f1e42699f4e40da65bbc33d
I20240426 09:53:56.849170 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 09:53:56.849524 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:53:56.893779 9170 raft_consensus.cc:979] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3: : Instructing follower cdbe92f5959a47dda817bbec2148eab9 to start an election
I20240426 09:53:56.894145 9222 raft_consensus.cc:1067] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 LEADER]: Signalling peer cdbe92f5959a47dda817bbec2148eab9 to start an election
I20240426 09:53:56.895339 8891 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009"
dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
from {username='slave'} at 127.0.0.1:53924
I20240426 09:53:56.895722 8891 raft_consensus.cc:482] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:56.895951 8891 raft_consensus.cc:3037] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:53:56.899931 8891 raft_consensus.cc:504] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:56.901645 8891 leader_election.cc:288] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 4 election: Requested vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:56.902376 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:53:56.902554 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:56.902885 8968 raft_consensus.cc:3032] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:53:56.903035 9042 raft_consensus.cc:3037] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:53:56.903144 8968 raft_consensus.cc:729] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:56.903800 8968 consensus_queue.cc:260] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:56.904861 8968 raft_consensus.cc:3037] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:53:56.907696 9042 raft_consensus.cc:2454] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 4.
I20240426 09:53:56.908576 8859 leader_election.cc:302] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdbe92f5959a47dda817bbec2148eab9, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:56.908759 8968 raft_consensus.cc:2454] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 4.
I20240426 09:53:56.909853 9223 raft_consensus.cc:2781] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:53:56.910517 9223 raft_consensus.cc:686] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 4 LEADER]: Becoming Leader. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Running, Role: LEADER
I20240426 09:53:56.911271 9223 consensus_queue.cc:237] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:56.919507 8787 catalog_manager.cc:5484] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 reported cstate change: term changed from 3 to 4, leader changed from 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130) to cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129). New cstate: current_term: 4 leader_uuid: "cdbe92f5959a47dda817bbec2148eab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:56.952365 9222 raft_consensus.cc:979] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3: : Instructing follower cdbe92f5959a47dda817bbec2148eab9 to start an election
I20240426 09:53:56.952723 9170 raft_consensus.cc:1067] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 LEADER]: Signalling peer cdbe92f5959a47dda817bbec2148eab9 to start an election
I20240426 09:53:56.953807 8891 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73"
dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
from {username='slave'} at 127.0.0.1:53924
I20240426 09:53:56.954226 8891 raft_consensus.cc:482] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:56.954460 8891 raft_consensus.cc:3037] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:53:56.958036 8891 raft_consensus.cc:504] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:56.959236 8891 leader_election.cc:288] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 4 election: Requested vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:56.959885 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:53:56.960104 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:56.960332 8968 raft_consensus.cc:3032] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:53:56.960533 9042 raft_consensus.cc:3037] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:53:56.960572 8968 raft_consensus.cc:729] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:56.961163 8968 consensus_queue.cc:260] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:56.961905 8968 raft_consensus.cc:3037] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:53:56.964021 9042 raft_consensus.cc:2454] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 4.
I20240426 09:53:56.964742 8859 leader_election.cc:302] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdbe92f5959a47dda817bbec2148eab9, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:56.965294 9223 raft_consensus.cc:2781] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:53:56.965528 8968 raft_consensus.cc:2454] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 4.
I20240426 09:53:56.965636 9223 raft_consensus.cc:686] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 4 LEADER]: Becoming Leader. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Running, Role: LEADER
I20240426 09:53:56.966624 9223 consensus_queue.cc:237] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:56.972378 8787 catalog_manager.cc:5484] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 reported cstate change: term changed from 3 to 4, leader changed from 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130) to cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129). New cstate: current_term: 4 leader_uuid: "cdbe92f5959a47dda817bbec2148eab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:53:57.076023 9222 raft_consensus.cc:979] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3: : Instructing follower cdbe92f5959a47dda817bbec2148eab9 to start an election
I20240426 09:53:57.076378 9170 raft_consensus.cc:1067] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 LEADER]: Signalling peer cdbe92f5959a47dda817bbec2148eab9 to start an election
I20240426 09:53:57.077502 8891 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "431d4df7c67747fcb0812c91964c801b"
dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
from {username='slave'} at 127.0.0.1:53924
I20240426 09:53:57.077884 8891 raft_consensus.cc:482] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:57.078128 8891 raft_consensus.cc:3037] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:57.081749 8891 raft_consensus.cc:504] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:57.083050 8891 leader_election.cc:288] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 3 election: Requested vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:53:57.083724 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431d4df7c67747fcb0812c91964c801b" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:53:57.083917 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431d4df7c67747fcb0812c91964c801b" candidate_uuid: "cdbe92f5959a47dda817bbec2148eab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:53:57.084132 8968 raft_consensus.cc:3032] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:53:57.084354 9042 raft_consensus.cc:3037] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:57.084386 8968 raft_consensus.cc:729] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:57.084985 8968 consensus_queue.cc:260] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:57.085793 8968 raft_consensus.cc:3037] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:57.089098 9042 raft_consensus.cc:2454] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 3.
I20240426 09:53:57.089911 8859 leader_election.cc:302] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdbe92f5959a47dda817bbec2148eab9, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:57.090497 9223 raft_consensus.cc:2781] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:53:57.090891 8968 raft_consensus.cc:2454] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbe92f5959a47dda817bbec2148eab9 in term 3.
I20240426 09:53:57.090938 9223 raft_consensus.cc:686] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 3 LEADER]: Becoming Leader. State: Replica: cdbe92f5959a47dda817bbec2148eab9, State: Running, Role: LEADER
I20240426 09:53:57.091885 9223 consensus_queue.cc:237] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:57.097448 8787 catalog_manager.cc:5484] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 reported cstate change: term changed from 2 to 3, leader changed from 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130) to cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129). New cstate: current_term: 3 leader_uuid: "cdbe92f5959a47dda817bbec2148eab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
W20240426 09:53:57.188935 8821 debug-util.cc:398] Leaking SignalData structure 0x7b0800061060 after lost signal to thread 8761
W20240426 09:53:57.189644 8821 debug-util.cc:398] Leaking SignalData structure 0x7b0800061740 after lost signal to thread 8824
W20240426 09:53:57.190341 8821 debug-util.cc:398] Leaking SignalData structure 0x7b08001ece60 after lost signal to thread 8917
W20240426 09:53:57.191008 8821 debug-util.cc:398] Leaking SignalData structure 0x7b08001fad20 after lost signal to thread 8992
W20240426 09:53:57.191613 8821 debug-util.cc:398] Leaking SignalData structure 0x7b08001c1940 after lost signal to thread 9066
I20240426 09:53:57.312476 8968 raft_consensus.cc:1261] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Refusing update from remote peer cdbe92f5959a47dda817bbec2148eab9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:53:57.313720 9223 consensus_queue.cc:1035] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:53:57.315666 9222 raft_consensus.cc:979] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3: : Instructing follower d99df4d00f1e42699f4e40da65bbc33d to start an election
I20240426 09:53:57.316090 9222 raft_consensus.cc:1067] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 LEADER]: Signalling peer d99df4d00f1e42699f4e40da65bbc33d to start an election
I20240426 09:53:57.317708 9042 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97deaa8654f64ad2a63314ffaa91d955"
dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
from {username='slave'} at 127.0.0.1:37844
I20240426 09:53:57.317893 9170 raft_consensus.cc:979] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3: : Instructing follower d99df4d00f1e42699f4e40da65bbc33d to start an election
I20240426 09:53:57.318326 9170 raft_consensus.cc:1067] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 LEADER]: Signalling peer d99df4d00f1e42699f4e40da65bbc33d to start an election
I20240426 09:53:57.318236 9042 raft_consensus.cc:482] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:57.318882 9042 raft_consensus.cc:3037] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:57.319985 9041 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec"
dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
from {username='slave'} at 127.0.0.1:37844
I20240426 09:53:57.320549 9041 raft_consensus.cc:482] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:53:57.320884 9041 raft_consensus.cc:3037] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:53:57.325359 9042 raft_consensus.cc:504] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:57.328451 8891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97deaa8654f64ad2a63314ffaa91d955" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
I20240426 09:53:57.328974 8891 raft_consensus.cc:3037] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:57.335161 9040 raft_consensus.cc:1261] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Refusing update from remote peer cdbe92f5959a47dda817bbec2148eab9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:53:57.336521 9223 consensus_queue.cc:1035] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:53:57.336892 8891 raft_consensus.cc:2454] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 3.
I20240426 09:53:57.336985 9041 raft_consensus.cc:504] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:57.338197 9007 leader_election.cc:302] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdbe92f5959a47dda817bbec2148eab9, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:57.338917 9194 raft_consensus.cc:2781] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:53:57.340081 9042 leader_election.cc:288] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 3 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991)
I20240426 09:53:57.341053 8966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:53:57.341599 8966 raft_consensus.cc:3032] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:53:57.344812 8891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c779579a0648ee982d552a8fec0bec" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "cdbe92f5959a47dda817bbec2148eab9"
I20240426 09:53:57.345322 8891 raft_consensus.cc:3037] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:53:57.341884 8966 raft_consensus.cc:729] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:57.351277 8891 raft_consensus.cc:2454] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 5.
I20240426 09:53:57.344197 9194 raft_consensus.cc:686] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 3 LEADER]: Becoming Leader. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Running, Role: LEADER
I20240426 09:53:57.351912 8967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97deaa8654f64ad2a63314ffaa91d955" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:53:57.352465 8967 raft_consensus.cc:3032] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:53:57.352793 8967 raft_consensus.cc:729] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:53:57.352324 9194 consensus_queue.cc:237] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:57.353549 8967 consensus_queue.cc:260] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:57.343535 9041 leader_election.cc:288] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 5 election: Requested vote from peers 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991), cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507)
I20240426 09:53:57.355535 8966 consensus_queue.cc:260] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:57.354647 8967 raft_consensus.cc:3037] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:53:57.359972 8966 raft_consensus.cc:3037] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:53:57.367223 8967 raft_consensus.cc:2454] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 3.
I20240426 09:53:57.368243 9007 leader_election.cc:302] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdbe92f5959a47dda817bbec2148eab9, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:53:57.370225 9235 raft_consensus.cc:2781] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:53:57.370743 9235 raft_consensus.cc:686] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 5 LEADER]: Becoming Leader. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Running, Role: LEADER
I20240426 09:53:57.371686 9235 consensus_queue.cc:237] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:53:57.381448 8966 raft_consensus.cc:2454] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 5.
I20240426 09:53:57.409489 8787 catalog_manager.cc:5484] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d reported cstate change: term changed from 2 to 3, leader changed from 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130) to d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131). New cstate: current_term: 3 leader_uuid: "d99df4d00f1e42699f4e40da65bbc33d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: HEALTHY } } }
I20240426 09:53:57.430359 8787 catalog_manager.cc:5484] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d reported cstate change: term changed from 4 to 5, leader changed from 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130) to d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131). New cstate: current_term: 5 leader_uuid: "d99df4d00f1e42699f4e40da65bbc33d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: HEALTHY } } }
I20240426 09:53:57.486146 9040 raft_consensus.cc:1261] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Refusing update from remote peer cdbe92f5959a47dda817bbec2148eab9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:53:57.487499 9223 consensus_queue.cc:1035] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 09:53:57.490832 8966 raft_consensus.cc:1261] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Refusing update from remote peer cdbe92f5959a47dda817bbec2148eab9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:53:57.492218 9224 consensus_queue.cc:1035] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 09:53:57.574515 8966 raft_consensus.cc:1261] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Refusing update from remote peer cdbe92f5959a47dda817bbec2148eab9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:53:57.575672 9244 consensus_queue.cc:1035] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 09:53:57.587654 9040 raft_consensus.cc:1261] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Refusing update from remote peer cdbe92f5959a47dda817bbec2148eab9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:53:57.589689 9244 consensus_queue.cc:1035] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:53:57.769800 8891 raft_consensus.cc:1261] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 5 FOLLOWER]: Refusing update from remote peer d99df4d00f1e42699f4e40da65bbc33d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240426 09:53:57.770511 8892 raft_consensus.cc:1261] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Refusing update from remote peer d99df4d00f1e42699f4e40da65bbc33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:53:57.771497 9235 consensus_queue.cc:1035] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 09:53:57.772792 9235 consensus_queue.cc:1035] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:53:57.787045 8966 raft_consensus.cc:1261] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 5 FOLLOWER]: Refusing update from remote peer d99df4d00f1e42699f4e40da65bbc33d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240426 09:53:57.789314 9235 consensus_queue.cc:1035] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:53:57.793112 8966 raft_consensus.cc:1261] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Refusing update from remote peer d99df4d00f1e42699f4e40da65bbc33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:53:57.795017 9235 consensus_queue.cc:1035] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:53:57.850288 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:53:57.855928 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:53:57.856277 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:53:58.856933 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:53:58.861513 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:53:58.861892 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:53:59.862532 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:53:59.867074 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:53:59.867408 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:00.868044 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:00.872570 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:00.872911 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:01.873625 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:01.878247 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:01.878614 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:02.883256 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:02.888433 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:02.888922 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:03.889648 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:03.894114 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:03.894421 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:04.895047 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:04.899706 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:04.900039 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:06.006299 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:06.011108 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:06.011431 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240426 09:54:06.488834 8989 debug-util.cc:398] Leaking SignalData structure 0x7b0800134060 after lost signal to thread 8761
W20240426 09:54:06.489462 8989 debug-util.cc:398] Leaking SignalData structure 0x7b0800134740 after lost signal to thread 8824
W20240426 09:54:06.490100 8989 debug-util.cc:398] Leaking SignalData structure 0x7b080017bec0 after lost signal to thread 8917
W20240426 09:54:06.490681 8989 debug-util.cc:398] Leaking SignalData structure 0x7b08001e19e0 after lost signal to thread 8992
W20240426 09:54:06.491307 8989 debug-util.cc:398] Leaking SignalData structure 0x7b08001f7100 after lost signal to thread 9066
I20240426 09:54:07.106730 8758 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20240426 09:54:07.107498 8758 tablet_server.cc:178] TabletServer@127.8.141.129:0 shutting down...
W20240426 09:54:07.117928 8933 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240426 09:54:07.121984 8933 consensus_peers.cc:485] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 -> Peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Couldn't send request to peer cdbe92f5959a47dda817bbec2148eab9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:07.141431 8758 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 09:54:07.142179 8758 tablet_replica.cc:330] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9: stopping tablet replica
I20240426 09:54:07.143152 8758 raft_consensus.cc:2229] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 4 LEADER]: Raft consensus shutting down.
I20240426 09:54:07.144132 8758 raft_consensus.cc:2258] T 69d609ac70eb4a09b5bc56973c5d3009 P cdbe92f5959a47dda817bbec2148eab9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:07.147136 8758 tablet_replica.cc:330] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9: stopping tablet replica
I20240426 09:54:07.147747 8758 raft_consensus.cc:2229] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:07.148257 8758 raft_consensus.cc:2258] T 4af8a55ea6e5429093a5ec516bd3f6a0 P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:07.150302 8758 tablet_replica.cc:330] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9: stopping tablet replica
I20240426 09:54:07.151019 8758 raft_consensus.cc:2229] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:07.151495 8758 raft_consensus.cc:2258] T 91ee2df31bb64b6a9536f91149a87c0d P cdbe92f5959a47dda817bbec2148eab9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:07.153726 8758 tablet_replica.cc:330] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9: stopping tablet replica
I20240426 09:54:07.154426 8758 raft_consensus.cc:2229] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:07.155045 8758 raft_consensus.cc:2258] T df73379416114f8ab7efd3caf9c22e7e P cdbe92f5959a47dda817bbec2148eab9 [term 4 FOLLOWER]: Raft consensus is shut down!
W20240426 09:54:07.155220 9007 consensus_peers.cc:485] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d -> Peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Couldn't send request to peer cdbe92f5959a47dda817bbec2148eab9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:07.157132 8758 tablet_replica.cc:330] T 7b463d0d799d4c77abcbb023cc934bf2 P cdbe92f5959a47dda817bbec2148eab9: stopping tablet replica
I20240426 09:54:07.157778 8758 raft_consensus.cc:2229] T 7b463d0d799d4c77abcbb023cc934bf2 P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:07.158214 8758 raft_consensus.cc:2258] T 7b463d0d799d4c77abcbb023cc934bf2 P cdbe92f5959a47dda817bbec2148eab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:07.160349 8758 tablet_replica.cc:330] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9: stopping tablet replica
I20240426 09:54:07.160969 8758 raft_consensus.cc:2229] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:07.161423 8758 raft_consensus.cc:2258] T 97deaa8654f64ad2a63314ffaa91d955 P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:07.163373 8758 tablet_replica.cc:330] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9: stopping tablet replica
I20240426 09:54:07.163981 8758 raft_consensus.cc:2229] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 4 LEADER]: Raft consensus shutting down.
I20240426 09:54:07.164757 8758 raft_consensus.cc:2258] T 1dfc67501c0a441d9ac98720108b6f73 P cdbe92f5959a47dda817bbec2148eab9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:07.166886 8758 tablet_replica.cc:330] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9: stopping tablet replica
I20240426 09:54:07.167460 8758 raft_consensus.cc:2229] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:07.167981 8758 raft_consensus.cc:2258] T 38c779579a0648ee982d552a8fec0bec P cdbe92f5959a47dda817bbec2148eab9 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:07.169992 8758 tablet_replica.cc:330] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9: stopping tablet replica
I20240426 09:54:07.170607 8758 raft_consensus.cc:2229] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 3 LEADER]: Raft consensus shutting down.
I20240426 09:54:07.171523 8758 raft_consensus.cc:2258] T 431d4df7c67747fcb0812c91964c801b P cdbe92f5959a47dda817bbec2148eab9 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240426 09:54:07.323971 8914 debug-util.cc:398] Leaking SignalData structure 0x7b08000ec060 after lost signal to thread 8761
W20240426 09:54:07.324604 8914 debug-util.cc:398] Leaking SignalData structure 0x7b08000ec740 after lost signal to thread 8824
W20240426 09:54:07.325378 8914 debug-util.cc:398] Leaking SignalData structure 0x7b08000c4de0 after lost signal to thread 8917
W20240426 09:54:07.326062 8914 debug-util.cc:398] Leaking SignalData structure 0x7b0800111d80 after lost signal to thread 8992
W20240426 09:54:07.326895 8914 debug-util.cc:398] Leaking SignalData structure 0x7b080011cdc0 after lost signal to thread 9066
W20240426 09:54:07.368583 9007 consensus_peers.cc:485] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d -> Peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Couldn't send request to peer cdbe92f5959a47dda817bbec2148eab9. Status: Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 09:54:07.398507 8933 consensus_peers.cc:485] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 -> Peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Couldn't send request to peer cdbe92f5959a47dda817bbec2148eab9. Status: Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 09:54:07.516861 9007 consensus_peers.cc:485] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d -> Peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Couldn't send request to peer cdbe92f5959a47dda817bbec2148eab9. Status: Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 09:54:07.524336 8933 consensus_peers.cc:485] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 -> Peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Couldn't send request to peer cdbe92f5959a47dda817bbec2148eab9. Status: Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:08.091003 9267 raft_consensus.cc:482] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cdbe92f5959a47dda817bbec2148eab9)
I20240426 09:54:08.091763 9267 raft_consensus.cc:504] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:54:08.094461 9267 leader_election.cc:288] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991)
I20240426 09:54:08.103652 8965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3" is_pre_election: true
W20240426 09:54:08.108772 9007 leader_election.cc:334] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111)
I20240426 09:54:08.109217 9007 leader_election.cc:302] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d99df4d00f1e42699f4e40da65bbc33d; no voters: 9e98a77341c74d6d90d01cd3e39586b3, cdbe92f5959a47dda817bbec2148eab9
I20240426 09:54:08.110468 9266 raft_consensus.cc:2726] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:54:08.223263 9269 raft_consensus.cc:482] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cdbe92f5959a47dda817bbec2148eab9)
I20240426 09:54:08.223946 9269 raft_consensus.cc:504] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:54:08.227687 9040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431d4df7c67747fcb0812c91964c801b" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d" is_pre_election: true
I20240426 09:54:08.228966 9269 leader_election.cc:288] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
W20240426 09:54:08.230862 8933 leader_election.cc:334] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111)
I20240426 09:54:08.231267 8933 leader_election.cc:302] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3; no voters: cdbe92f5959a47dda817bbec2148eab9, d99df4d00f1e42699f4e40da65bbc33d
I20240426 09:54:08.231938 9269 raft_consensus.cc:2726] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 09:54:08.255857 8758 thread.cc:536] Waited for 1000ms trying to join with diag-logger (tid 8914)
I20240426 09:54:08.343086 9269 raft_consensus.cc:482] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cdbe92f5959a47dda817bbec2148eab9)
I20240426 09:54:08.343755 9269 raft_consensus.cc:504] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:54:08.346899 9040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d" is_pre_election: true
I20240426 09:54:08.348462 9269 leader_election.cc:288] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
W20240426 09:54:08.349895 8933 leader_election.cc:334] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111)
I20240426 09:54:08.350306 8933 leader_election.cc:302] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3; no voters: cdbe92f5959a47dda817bbec2148eab9, d99df4d00f1e42699f4e40da65bbc33d
I20240426 09:54:08.351002 9269 raft_consensus.cc:2726] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:54:08.394006 9269 raft_consensus.cc:482] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cdbe92f5959a47dda817bbec2148eab9)
I20240426 09:54:08.394560 9269 raft_consensus.cc:504] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:54:08.397491 9040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d" is_pre_election: true
I20240426 09:54:08.398115 9040 raft_consensus.cc:2454] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 4.
I20240426 09:54:08.399118 8934 leader_election.cc:302] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters:
I20240426 09:54:08.399953 9268 raft_consensus.cc:2781] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:54:08.400279 9268 raft_consensus.cc:482] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Starting leader election (detected failure of leader cdbe92f5959a47dda817bbec2148eab9)
I20240426 09:54:08.400595 9268 raft_consensus.cc:3037] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Advancing to term 5
W20240426 09:54:08.400719 8933 leader_election.cc:334] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111)
I20240426 09:54:08.401713 9269 leader_election.cc:288] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:54:08.407974 9268 raft_consensus.cc:504] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:54:08.410740 9040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d609ac70eb4a09b5bc56973c5d3009" candidate_uuid: "9e98a77341c74d6d90d01cd3e39586b3" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d99df4d00f1e42699f4e40da65bbc33d"
I20240426 09:54:08.411302 9040 raft_consensus.cc:3037] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Advancing to term 5
W20240426 09:54:08.413627 8933 leader_election.cc:334] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111)
I20240426 09:54:08.414281 9268 leader_election.cc:288] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 5 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131:43019)
I20240426 09:54:08.417392 9040 raft_consensus.cc:2454] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e98a77341c74d6d90d01cd3e39586b3 in term 5.
I20240426 09:54:08.418294 8934 leader_election.cc:302] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters: cdbe92f5959a47dda817bbec2148eab9
I20240426 09:54:08.418987 9268 raft_consensus.cc:2781] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:54:08.419394 9266 raft_consensus.cc:482] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cdbe92f5959a47dda817bbec2148eab9)
I20240426 09:54:08.419494 9268 raft_consensus.cc:686] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 5 LEADER]: Becoming Leader. State: Replica: 9e98a77341c74d6d90d01cd3e39586b3, State: Running, Role: LEADER
I20240426 09:54:08.419945 9266 raft_consensus.cc:504] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:54:08.420523 9268 consensus_queue.cc:237] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:54:08.421862 9266 leader_election.cc:288] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991)
I20240426 09:54:08.425516 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3" is_pre_election: true
I20240426 09:54:08.426082 8968 raft_consensus.cc:2454] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 4.
I20240426 09:54:08.427032 9007 leader_election.cc:302] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters:
W20240426 09:54:08.427702 9007 leader_election.cc:334] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111)
I20240426 09:54:08.428522 9266 raft_consensus.cc:2781] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:54:08.428959 9266 raft_consensus.cc:482] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Starting leader election (detected failure of leader cdbe92f5959a47dda817bbec2148eab9)
I20240426 09:54:08.429314 9266 raft_consensus.cc:3037] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:54:08.435586 8788 catalog_manager.cc:5484] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 reported cstate change: term changed from 4 to 5, leader changed from cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129) to 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130). New cstate: current_term: 5 leader_uuid: "9e98a77341c74d6d90d01cd3e39586b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:08.438634 9266 raft_consensus.cc:504] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:54:08.440508 9266 leader_election.cc:288] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 5 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991)
I20240426 09:54:08.443130 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfc67501c0a441d9ac98720108b6f73" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:54:08.443639 8968 raft_consensus.cc:3037] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Advancing to term 5
W20240426 09:54:08.452095 9007 leader_election.cc:334] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111)
I20240426 09:54:08.453104 8968 raft_consensus.cc:2454] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 5.
I20240426 09:54:08.454309 9007 leader_election.cc:302] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters: cdbe92f5959a47dda817bbec2148eab9
I20240426 09:54:08.455137 9266 raft_consensus.cc:2781] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:54:08.455555 9266 raft_consensus.cc:686] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 5 LEADER]: Becoming Leader. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Running, Role: LEADER
I20240426 09:54:08.456355 9266 consensus_queue.cc:237] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:54:08.474701 8790 catalog_manager.cc:5484] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d reported cstate change: term changed from 4 to 5, leader changed from cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129) to d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131). New cstate: current_term: 5 leader_uuid: "d99df4d00f1e42699f4e40da65bbc33d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:08.629741 9266 raft_consensus.cc:482] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cdbe92f5959a47dda817bbec2148eab9)
I20240426 09:54:08.630277 9266 raft_consensus.cc:504] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:54:08.634384 9266 leader_election.cc:288] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991)
I20240426 09:54:08.635850 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431d4df7c67747fcb0812c91964c801b" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3" is_pre_election: true
W20240426 09:54:08.636014 9007 leader_election.cc:334] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111)
I20240426 09:54:08.636574 8968 raft_consensus.cc:2454] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 3.
I20240426 09:54:08.637569 9007 leader_election.cc:302] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters: cdbe92f5959a47dda817bbec2148eab9
I20240426 09:54:08.639046 9266 raft_consensus.cc:2781] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:54:08.639374 9266 raft_consensus.cc:482] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Starting leader election (detected failure of leader cdbe92f5959a47dda817bbec2148eab9)
I20240426 09:54:08.639760 9266 raft_consensus.cc:3037] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:54:08.648938 9266 raft_consensus.cc:504] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:54:08.650684 9266 leader_election.cc:288] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 4 election: Requested vote from peers cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507), 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991)
I20240426 09:54:08.652752 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431d4df7c67747fcb0812c91964c801b" candidate_uuid: "d99df4d00f1e42699f4e40da65bbc33d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9e98a77341c74d6d90d01cd3e39586b3"
I20240426 09:54:08.653312 8968 raft_consensus.cc:3037] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:54:08.654960 9007 leader_election.cc:334] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111)
I20240426 09:54:08.658996 8968 raft_consensus.cc:2454] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99df4d00f1e42699f4e40da65bbc33d in term 4.
I20240426 09:54:08.659859 9007 leader_election.cc:302] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9e98a77341c74d6d90d01cd3e39586b3, d99df4d00f1e42699f4e40da65bbc33d; no voters: cdbe92f5959a47dda817bbec2148eab9
I20240426 09:54:08.660598 9266 raft_consensus.cc:2781] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:54:08.661051 9266 raft_consensus.cc:686] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 4 LEADER]: Becoming Leader. State: Replica: d99df4d00f1e42699f4e40da65bbc33d, State: Running, Role: LEADER
I20240426 09:54:08.661849 9266 consensus_queue.cc:237] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } }
I20240426 09:54:08.669610 8790 catalog_manager.cc:5484] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d reported cstate change: term changed from 3 to 4, leader changed from cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129) to d99df4d00f1e42699f4e40da65bbc33d (127.8.141.131). New cstate: current_term: 4 leader_uuid: "d99df4d00f1e42699f4e40da65bbc33d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbe92f5959a47dda817bbec2148eab9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:08.933717 8968 raft_consensus.cc:1261] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 5 FOLLOWER]: Refusing update from remote peer d99df4d00f1e42699f4e40da65bbc33d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240426 09:54:08.934958 9266 consensus_queue.cc:1035] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240426 09:54:08.955271 9007 consensus_peers.cc:485] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d -> Peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Couldn't send request to peer cdbe92f5959a47dda817bbec2148eab9. Status: Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:08.997097 9040 raft_consensus.cc:1261] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 5 FOLLOWER]: Refusing update from remote peer 9e98a77341c74d6d90d01cd3e39586b3: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240426 09:54:08.997983 9268 consensus_queue.cc:1035] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99df4d00f1e42699f4e40da65bbc33d" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240426 09:54:09.010314 8933 consensus_peers.cc:485] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 -> Peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Couldn't send request to peer cdbe92f5959a47dda817bbec2148eab9. Status: Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 09:54:09.045316 9063 debug-util.cc:398] Leaking SignalData structure 0x7b0800175060 after lost signal to thread 8761
W20240426 09:54:09.046183 9063 debug-util.cc:398] Leaking SignalData structure 0x7b0800175740 after lost signal to thread 8824
W20240426 09:54:09.047232 9063 debug-util.cc:398] Leaking SignalData structure 0x7b0800151b00 after lost signal to thread 8992
W20240426 09:54:09.048115 9063 debug-util.cc:398] Leaking SignalData structure 0x7b08001ed7c0 after lost signal to thread 9066
W20240426 09:54:09.071060 9007 consensus_peers.cc:485] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d -> Peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Couldn't send request to peer cdbe92f5959a47dda817bbec2148eab9. Status: Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:09.183571 8968 raft_consensus.cc:1261] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Refusing update from remote peer d99df4d00f1e42699f4e40da65bbc33d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:54:09.184746 9284 consensus_queue.cc:1035] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e98a77341c74d6d90d01cd3e39586b3" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240426 09:54:09.256325 8758 thread.cc:536] Waited for 2000ms trying to join with diag-logger (tid 8914)
W20240426 09:54:09.385072 9007 consensus_peers.cc:485] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d -> Peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Couldn't send request to peer cdbe92f5959a47dda817bbec2148eab9. Status: Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 09:54:10.064100 9007 consensus_peers.cc:485] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d -> Peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Couldn't send request to peer cdbe92f5959a47dda817bbec2148eab9. Status: Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 09:54:10.103124 8933 consensus_peers.cc:485] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 -> Peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Couldn't send request to peer cdbe92f5959a47dda817bbec2148eab9. Status: Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 09:54:10.256428 8758 tablet_server.cc:194] TabletServer@127.8.141.129:0 shutdown complete.
W20240426 09:54:10.260843 9007 consensus_peers.cc:485] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d -> Peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Couldn't send request to peer cdbe92f5959a47dda817bbec2148eab9. Status: Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 09:54:10.273085 8933 consensus_peers.cc:485] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 -> Peer cdbe92f5959a47dda817bbec2148eab9 (127.8.141.129:42507): Couldn't send request to peer cdbe92f5959a47dda817bbec2148eab9. Status: Network error: Client connection negotiation failed: client connection to 127.8.141.129:42507: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 09:54:10.280956 8758 tablet_server.cc:178] TabletServer@127.8.141.130:0 shutting down...
I20240426 09:54:10.307653 8758 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 09:54:10.308236 8758 tablet_replica.cc:330] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3: stopping tablet replica
I20240426 09:54:10.308816 8758 raft_consensus.cc:2229] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:10.309228 8758 raft_consensus.cc:2258] T 431d4df7c67747fcb0812c91964c801b P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:10.311184 8758 tablet_replica.cc:330] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3: stopping tablet replica
I20240426 09:54:10.311650 8758 raft_consensus.cc:2229] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 5 LEADER]: Raft consensus shutting down.
I20240426 09:54:10.312311 8758 raft_consensus.cc:2258] T 69d609ac70eb4a09b5bc56973c5d3009 P 9e98a77341c74d6d90d01cd3e39586b3 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:10.313779 8758 tablet_replica.cc:330] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3: stopping tablet replica
I20240426 09:54:10.314239 8758 raft_consensus.cc:2229] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 LEADER]: Raft consensus shutting down.
I20240426 09:54:10.314975 8758 raft_consensus.cc:2258] T 4af8a55ea6e5429093a5ec516bd3f6a0 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:10.316514 8758 tablet_replica.cc:330] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3: stopping tablet replica
I20240426 09:54:10.317157 8758 raft_consensus.cc:2229] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:10.317616 8758 raft_consensus.cc:2258] T 1dfc67501c0a441d9ac98720108b6f73 P 9e98a77341c74d6d90d01cd3e39586b3 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:10.319473 8758 tablet_replica.cc:330] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3: stopping tablet replica
I20240426 09:54:10.319986 8758 raft_consensus.cc:2229] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:10.320367 8758 raft_consensus.cc:2258] T 97deaa8654f64ad2a63314ffaa91d955 P 9e98a77341c74d6d90d01cd3e39586b3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:10.322031 8758 tablet_replica.cc:330] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3: stopping tablet replica
I20240426 09:54:10.322445 8758 raft_consensus.cc:2229] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:10.322861 8758 raft_consensus.cc:2258] T 38c779579a0648ee982d552a8fec0bec P 9e98a77341c74d6d90d01cd3e39586b3 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:10.324431 8758 tablet_replica.cc:330] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3: stopping tablet replica
I20240426 09:54:10.324892 8758 raft_consensus.cc:2229] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 LEADER]: Raft consensus shutting down.
I20240426 09:54:10.325551 8758 raft_consensus.cc:2258] T df73379416114f8ab7efd3caf9c22e7e P 9e98a77341c74d6d90d01cd3e39586b3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:10.327142 8758 tablet_replica.cc:330] T 7b463d0d799d4c77abcbb023cc934bf2 P 9e98a77341c74d6d90d01cd3e39586b3: stopping tablet replica
I20240426 09:54:10.327538 8758 raft_consensus.cc:2229] T 7b463d0d799d4c77abcbb023cc934bf2 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:10.327872 8758 raft_consensus.cc:2258] T 7b463d0d799d4c77abcbb023cc934bf2 P 9e98a77341c74d6d90d01cd3e39586b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:10.329331 8758 tablet_replica.cc:330] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3: stopping tablet replica
I20240426 09:54:10.329715 8758 raft_consensus.cc:2229] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 LEADER]: Raft consensus shutting down.
I20240426 09:54:10.330348 8758 raft_consensus.cc:2258] T 91ee2df31bb64b6a9536f91149a87c0d P 9e98a77341c74d6d90d01cd3e39586b3 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240426 09:54:10.342837 8989 debug-util.cc:398] Leaking SignalData structure 0x7b08001f70e0 after lost signal to thread 8761
W20240426 09:54:10.343627 8989 debug-util.cc:398] Leaking SignalData structure 0x7b08001f35e0 after lost signal to thread 8824
W20240426 09:54:10.344635 8989 debug-util.cc:398] Leaking SignalData structure 0x7b0800134100 after lost signal to thread 8992
W20240426 09:54:10.344839 9007 consensus_peers.cc:485] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d -> Peer 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991): Couldn't send request to peer 9e98a77341c74d6d90d01cd3e39586b3. Status: Network error: Client connection negotiation failed: client connection to 127.8.141.130:35991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 09:54:10.345470 8989 debug-util.cc:398] Leaking SignalData structure 0x7b08000a7ee0 after lost signal to thread 9066
W20240426 09:54:10.368546 9007 consensus_peers.cc:485] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d -> Peer 9e98a77341c74d6d90d01cd3e39586b3 (127.8.141.130:35991): Couldn't send request to peer 9e98a77341c74d6d90d01cd3e39586b3. Status: Network error: Client connection negotiation failed: client connection to 127.8.141.130:35991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:10.691237 8758 tablet_server.cc:194] TabletServer@127.8.141.130:0 shutdown complete.
I20240426 09:54:10.709841 8758 tablet_server.cc:178] TabletServer@127.8.141.131:0 shutting down...
I20240426 09:54:10.737352 8758 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 09:54:10.737953 8758 tablet_replica.cc:330] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d: stopping tablet replica
I20240426 09:54:10.738680 8758 raft_consensus.cc:2229] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 4 LEADER]: Raft consensus shutting down.
I20240426 09:54:10.739614 8758 raft_consensus.cc:2258] T 431d4df7c67747fcb0812c91964c801b P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:10.741215 8758 tablet_replica.cc:330] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d: stopping tablet replica
I20240426 09:54:10.741746 8758 raft_consensus.cc:2229] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 3 LEADER]: Raft consensus shutting down.
I20240426 09:54:10.742336 8758 raft_consensus.cc:2258] T 97deaa8654f64ad2a63314ffaa91d955 P d99df4d00f1e42699f4e40da65bbc33d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:10.743893 8758 tablet_replica.cc:330] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d: stopping tablet replica
I20240426 09:54:10.744333 8758 raft_consensus.cc:2229] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 5 LEADER]: Raft consensus shutting down.
I20240426 09:54:10.744916 8758 raft_consensus.cc:2258] T 1dfc67501c0a441d9ac98720108b6f73 P d99df4d00f1e42699f4e40da65bbc33d [term 5 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:10.746963 8758 tablet_replica.cc:330] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d: stopping tablet replica
I20240426 09:54:10.747660 8758 raft_consensus.cc:2229] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 5 LEADER]: Raft consensus shutting down.
I20240426 09:54:10.748499 8758 raft_consensus.cc:2258] T 38c779579a0648ee982d552a8fec0bec P d99df4d00f1e42699f4e40da65bbc33d [term 5 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:10.750761 8758 tablet_replica.cc:330] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d: stopping tablet replica
I20240426 09:54:10.751461 8758 raft_consensus.cc:2229] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 5 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:10.751971 8758 raft_consensus.cc:2258] T 69d609ac70eb4a09b5bc56973c5d3009 P d99df4d00f1e42699f4e40da65bbc33d [term 5 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:10.753571 8758 tablet_replica.cc:330] T 91ee2df31bb64b6a9536f91149a87c0d P d99df4d00f1e42699f4e40da65bbc33d: stopping tablet replica
I20240426 09:54:10.753999 8758 raft_consensus.cc:2229] T 91ee2df31bb64b6a9536f91149a87c0d P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:10.754354 8758 raft_consensus.cc:2258] T 91ee2df31bb64b6a9536f91149a87c0d P d99df4d00f1e42699f4e40da65bbc33d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:10.756032 8758 tablet_replica.cc:330] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d: stopping tablet replica
I20240426 09:54:10.756462 8758 raft_consensus.cc:2229] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:10.756880 8758 raft_consensus.cc:2258] T df73379416114f8ab7efd3caf9c22e7e P d99df4d00f1e42699f4e40da65bbc33d [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:10.758927 8758 tablet_replica.cc:330] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d: stopping tablet replica
I20240426 09:54:10.759459 8758 raft_consensus.cc:2229] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [term 1 LEADER]: Raft consensus shutting down.
I20240426 09:54:10.760118 8758 raft_consensus.cc:2258] T 7b463d0d799d4c77abcbb023cc934bf2 P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:10.762107 8758 tablet_replica.cc:330] T 4af8a55ea6e5429093a5ec516bd3f6a0 P d99df4d00f1e42699f4e40da65bbc33d: stopping tablet replica
I20240426 09:54:10.762547 8758 raft_consensus.cc:2229] T 4af8a55ea6e5429093a5ec516bd3f6a0 P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:10.763006 8758 raft_consensus.cc:2258] T 4af8a55ea6e5429093a5ec516bd3f6a0 P d99df4d00f1e42699f4e40da65bbc33d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:11.434721 8758 tablet_server.cc:194] TabletServer@127.8.141.131:0 shutdown complete.
I20240426 09:54:11.449793 8758 master.cc:537] Master@127.8.141.190:37867 shutting down...
I20240426 09:54:11.480314 8845 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Closing
I20240426 09:54:11.555437 8758 raft_consensus.cc:2229] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [term 1 LEADER]: Raft consensus shutting down.
I20240426 09:54:11.555915 8758 raft_consensus.cc:2258] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:11.556185 8758 tablet_replica.cc:330] T 00000000000000000000000000000000 P af87027f7d3c4ebfb3d5a5daa8cfc402: stopping tablet replica
I20240426 09:54:11.587877 8758 master.cc:552] Master@127.8.141.190:37867 shutdown complete.
9e98a77341c74d6d90d01cd3e39586b3 4
cdbe92f5959a47dda817bbec2148eab9 4
d99df4d00f1e42699f4e40da65bbc33d 1
9e98a77341c74d6d90d01cd3e39586b3 3
cdbe92f5959a47dda817bbec2148eab9 3
d99df4d00f1e42699f4e40da65bbc33d 3
9e98a77341c74d6d90d01cd3e39586b3 8
cdbe92f5959a47dda817bbec2148eab9 0
d99df4d00f1e42699f4e40da65bbc33d 1
9e98a77341c74d6d90d01cd3e39586b3 3
cdbe92f5959a47dda817bbec2148eab9 3
d99df4d00f1e42699f4e40da65bbc33d 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (34319 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20240426 09:54:11.624637 8758 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.8.141.190:35665
I20240426 09:54:11.625656 8758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:54:11.631801 9293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:54:11.632301 9294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:54:11.632725 9296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:54:11.633293 8758 server_base.cc:1034] running on GCE node
I20240426 09:54:11.634192 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 09:54:11.634366 8758 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 09:54:11.634506 8758 hybrid_clock.cc:648] HybridClock initialized: now 1714125251634491 us; error 0 us; skew 500 ppm
I20240426 09:54:11.635030 8758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:54:11.640777 8758 webserver.cc:460] Webserver started at http://127.8.141.190:38215/ using document root <none> and password file <none>
I20240426 09:54:11.641211 8758 fs_manager.cc:359] Metadata directory not provided
I20240426 09:54:11.641372 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:54:11.641628 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:54:11.642674 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/master-0-root/instance:
uuid: "aa415a75df1d436aac70627c304b84aa"
format_stamp: "Formatted at 2024-04-26 09:54:11 on dist-test-slave-5txv"
I20240426 09:54:11.646675 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240426 09:54:11.649665 9301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:11.650302 8758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240426 09:54:11.650537 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/master-0-root
uuid: "aa415a75df1d436aac70627c304b84aa"
format_stamp: "Formatted at 2024-04-26 09:54:11 on dist-test-slave-5txv"
I20240426 09:54:11.650817 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:54:11.666167 8758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 09:54:11.667251 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:54:11.700903 8758 rpc_server.cc:307] RPC server started. Bound to: 127.8.141.190:35665
I20240426 09:54:11.700981 9352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.141.190:35665 every 8 connection(s)
I20240426 09:54:11.704386 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:11.712833 9353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa: Bootstrap starting.
I20240426 09:54:11.716881 9353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:11.720546 9353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa: No bootstrap required, opened a new log
I20240426 09:54:11.722281 9353 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa415a75df1d436aac70627c304b84aa" member_type: VOTER }
I20240426 09:54:11.722637 9353 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:11.722888 9353 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa415a75df1d436aac70627c304b84aa, State: Initialized, Role: FOLLOWER
I20240426 09:54:11.723419 9353 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa415a75df1d436aac70627c304b84aa" member_type: VOTER }
I20240426 09:54:11.723837 9353 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240426 09:54:11.724043 9353 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240426 09:54:11.724287 9353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:11.727808 9353 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa415a75df1d436aac70627c304b84aa" member_type: VOTER }
I20240426 09:54:11.728274 9353 leader_election.cc:302] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa415a75df1d436aac70627c304b84aa; no voters:
I20240426 09:54:11.729233 9353 leader_election.cc:288] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20240426 09:54:11.729547 9356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:11.730644 9356 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [term 1 LEADER]: Becoming Leader. State: Replica: aa415a75df1d436aac70627c304b84aa, State: Running, Role: LEADER
I20240426 09:54:11.731278 9356 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa415a75df1d436aac70627c304b84aa" member_type: VOTER }
I20240426 09:54:11.731824 9353 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [sys.catalog]: configured and running, proceeding with master startup.
I20240426 09:54:11.735359 9357 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa415a75df1d436aac70627c304b84aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa415a75df1d436aac70627c304b84aa" member_type: VOTER } }
I20240426 09:54:11.735648 9358 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa415a75df1d436aac70627c304b84aa. Latest consensus state: current_term: 1 leader_uuid: "aa415a75df1d436aac70627c304b84aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa415a75df1d436aac70627c304b84aa" member_type: VOTER } }
I20240426 09:54:11.735960 9357 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [sys.catalog]: This master's current role is: LEADER
I20240426 09:54:11.736217 9358 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [sys.catalog]: This master's current role is: LEADER
I20240426 09:54:11.738574 9363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240426 09:54:11.743434 9363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240426 09:54:11.746068 8758 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240426 09:54:11.750914 9363 catalog_manager.cc:1347] Generated new cluster ID: 0cd75e4a8b244513bdee17e454670362
I20240426 09:54:11.751173 9363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240426 09:54:11.761468 9363 catalog_manager.cc:1370] Generated new certificate authority record
I20240426 09:54:11.762563 9363 catalog_manager.cc:1504] Loading token signing keys...
I20240426 09:54:11.778177 9363 catalog_manager.cc:5857] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa: Generated new TSK 0
I20240426 09:54:11.778699 9363 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240426 09:54:11.811686 8758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:54:11.817709 9374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:54:11.818310 9375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:54:11.819377 9377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:54:11.819972 8758 server_base.cc:1034] running on GCE node
I20240426 09:54:11.820770 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 09:54:11.820946 8758 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 09:54:11.821096 8758 hybrid_clock.cc:648] HybridClock initialized: now 1714125251821080 us; error 0 us; skew 500 ppm
I20240426 09:54:11.821588 8758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:54:11.824268 8758 webserver.cc:460] Webserver started at http://127.8.141.129:42973/ using document root <none> and password file <none>
I20240426 09:54:11.824765 8758 fs_manager.cc:359] Metadata directory not provided
I20240426 09:54:11.824945 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:54:11.825269 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:54:11.826270 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/ts-0-root/instance:
uuid: "a51b16892b1f45a6933030491fcc05aa"
format_stamp: "Formatted at 2024-04-26 09:54:11 on dist-test-slave-5txv"
I20240426 09:54:11.830236 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240426 09:54:11.833226 9382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:11.833863 8758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240426 09:54:11.834118 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/ts-0-root
uuid: "a51b16892b1f45a6933030491fcc05aa"
format_stamp: "Formatted at 2024-04-26 09:54:11 on dist-test-slave-5txv"
I20240426 09:54:11.834376 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:54:11.846244 8758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 09:54:11.847306 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:54:11.848654 8758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:54:11.850647 8758 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:54:11.850852 8758 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:11.851066 8758 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:54:11.851229 8758 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:11.887683 8758 rpc_server.cc:307] RPC server started. Bound to: 127.8.141.129:33159
I20240426 09:54:11.887764 9444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.141.129:33159 every 8 connection(s)
I20240426 09:54:11.891744 8758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:54:11.899487 9449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:54:11.900244 9450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:54:11.901541 8758 server_base.cc:1034] running on GCE node
W20240426 09:54:11.902292 9452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:54:11.903110 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 09:54:11.903313 8758 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 09:54:11.903486 8758 hybrid_clock.cc:648] HybridClock initialized: now 1714125251903467 us; error 0 us; skew 500 ppm
I20240426 09:54:11.903672 9445 heartbeater.cc:347] Connected to a master server at 127.8.141.190:35665
I20240426 09:54:11.903959 9445 heartbeater.cc:464] Registering TS with master...
I20240426 09:54:11.904042 8758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:54:11.904708 9445 heartbeater.cc:511] Master 127.8.141.190:35665 requested a full tablet report, sending...
I20240426 09:54:11.906442 9318 ts_manager.cc:194] Registered new tserver with Master: a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159)
I20240426 09:54:11.907212 8758 webserver.cc:460] Webserver started at http://127.8.141.130:33785/ using document root <none> and password file <none>
I20240426 09:54:11.907670 8758 fs_manager.cc:359] Metadata directory not provided
I20240426 09:54:11.907845 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:54:11.908075 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:54:11.908095 9318 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40706
I20240426 09:54:11.909111 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/ts-1-root/instance:
uuid: "88b95908620241b7b6992d56d159a653"
format_stamp: "Formatted at 2024-04-26 09:54:11 on dist-test-slave-5txv"
I20240426 09:54:11.912958 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240426 09:54:11.915843 9457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:11.916455 8758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240426 09:54:11.916698 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/ts-1-root
uuid: "88b95908620241b7b6992d56d159a653"
format_stamp: "Formatted at 2024-04-26 09:54:11 on dist-test-slave-5txv"
I20240426 09:54:11.916944 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:54:11.947851 8758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 09:54:11.949019 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:54:11.950321 8758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:54:11.952457 8758 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:54:11.952648 8758 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:11.952863 8758 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:54:11.953011 8758 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:11.988739 8758 rpc_server.cc:307] RPC server started. Bound to: 127.8.141.130:43215
I20240426 09:54:11.988838 9519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.141.130:43215 every 8 connection(s)
I20240426 09:54:11.992937 8758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:54:11.999420 9523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:54:12.000486 9524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:54:12.003835 9526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:54:12.004016 9520 heartbeater.cc:347] Connected to a master server at 127.8.141.190:35665
I20240426 09:54:12.004448 9520 heartbeater.cc:464] Registering TS with master...
I20240426 09:54:12.005051 9520 heartbeater.cc:511] Master 127.8.141.190:35665 requested a full tablet report, sending...
I20240426 09:54:12.005481 8758 server_base.cc:1034] running on GCE node
I20240426 09:54:12.006276 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 09:54:12.006510 8758 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 09:54:12.006672 8758 hybrid_clock.cc:648] HybridClock initialized: now 1714125252006654 us; error 0 us; skew 500 ppm
I20240426 09:54:12.006865 9318 ts_manager.cc:194] Registered new tserver with Master: 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215)
I20240426 09:54:12.007221 8758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:54:12.008073 9318 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40712
I20240426 09:54:12.009730 8758 webserver.cc:460] Webserver started at http://127.8.141.131:39527/ using document root <none> and password file <none>
I20240426 09:54:12.010205 8758 fs_manager.cc:359] Metadata directory not provided
I20240426 09:54:12.010385 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:54:12.010618 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:54:12.011667 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/ts-2-root/instance:
uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
format_stamp: "Formatted at 2024-04-26 09:54:12 on dist-test-slave-5txv"
I20240426 09:54:12.015623 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240426 09:54:12.018508 9531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:12.019177 8758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240426 09:54:12.019428 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/ts-2-root
uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
format_stamp: "Formatted at 2024-04-26 09:54:12 on dist-test-slave-5txv"
I20240426 09:54:12.019683 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714125217154154-8758-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:54:12.027868 8758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 09:54:12.028791 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:54:12.030016 8758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:54:12.032001 8758 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:54:12.032171 8758 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:12.032380 8758 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:54:12.032522 8758 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:12.069847 8758 rpc_server.cc:307] RPC server started. Bound to: 127.8.141.131:40209
I20240426 09:54:12.069947 9593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.141.131:40209 every 8 connection(s)
I20240426 09:54:12.081364 9594 heartbeater.cc:347] Connected to a master server at 127.8.141.190:35665
I20240426 09:54:12.081704 9594 heartbeater.cc:464] Registering TS with master...
I20240426 09:54:12.082376 9594 heartbeater.cc:511] Master 127.8.141.190:35665 requested a full tablet report, sending...
I20240426 09:54:12.084015 9318 ts_manager.cc:194] Registered new tserver with Master: 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.084549 8758 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.011844321s
I20240426 09:54:12.085227 9318 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40722
I20240426 09:54:12.086640 8758 test_util.cc:268] Using random seed: -490714278
I20240426 09:54:12.106236 9318 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40730:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240426 09:54:12.108909 9318 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240426 09:54:12.183522 9559 tablet_service.cc:1467] Processing CreateTablet for tablet 87f66b3dfc3d4b5abb5726efb3d225f1 (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240426 09:54:12.183763 9558 tablet_service.cc:1467] Processing CreateTablet for tablet 99873d472db84e08889ea87be1b4d6df (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240426 09:54:12.184921 9559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f66b3dfc3d4b5abb5726efb3d225f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.185715 9554 tablet_service.cc:1467] Processing CreateTablet for tablet b1990a88d033491bbf9904c47e91a23d (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240426 09:54:12.185904 9557 tablet_service.cc:1467] Processing CreateTablet for tablet 3db2ebff0e144fc1a1a5144e9c6f7e60 (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240426 09:54:12.186954 9554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1990a88d033491bbf9904c47e91a23d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.187593 9556 tablet_service.cc:1467] Processing CreateTablet for tablet b488843f1cb64132982e15fc409939fb (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240426 09:54:12.188702 9558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99873d472db84e08889ea87be1b4d6df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.189381 9555 tablet_service.cc:1467] Processing CreateTablet for tablet 9bbd3b59d2fc41d49d30fc63daf3d20c (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240426 09:54:12.191267 9550 tablet_service.cc:1467] Processing CreateTablet for tablet ff15116e7e7546b1b7b04c5b227e503e (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240426 09:54:12.193011 9551 tablet_service.cc:1467] Processing CreateTablet for tablet 32407ee797ba4359a5f79513e69741ad (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240426 09:54:12.194811 9552 tablet_service.cc:1467] Processing CreateTablet for tablet 313a150ad860483a978ae5584f5554e0 (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240426 09:54:12.196635 9553 tablet_service.cc:1467] Processing CreateTablet for tablet 75da9d0d79d24604a4bb97685008c10e (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240426 09:54:12.197932 9556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b488843f1cb64132982e15fc409939fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.199093 9555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bbd3b59d2fc41d49d30fc63daf3d20c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.200806 9551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32407ee797ba4359a5f79513e69741ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.201402 9552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313a150ad860483a978ae5584f5554e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.202545 9550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff15116e7e7546b1b7b04c5b227e503e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.207072 9557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db2ebff0e144fc1a1a5144e9c6f7e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.208644 9410 tablet_service.cc:1467] Processing CreateTablet for tablet 87f66b3dfc3d4b5abb5726efb3d225f1 (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240426 09:54:12.209805 9410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f66b3dfc3d4b5abb5726efb3d225f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.211266 9407 tablet_service.cc:1467] Processing CreateTablet for tablet b488843f1cb64132982e15fc409939fb (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240426 09:54:12.211885 9409 tablet_service.cc:1467] Processing CreateTablet for tablet 99873d472db84e08889ea87be1b4d6df (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240426 09:54:12.212462 9407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b488843f1cb64132982e15fc409939fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.213110 9409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99873d472db84e08889ea87be1b4d6df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.214375 9402 tablet_service.cc:1467] Processing CreateTablet for tablet 32407ee797ba4359a5f79513e69741ad (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240426 09:54:12.215554 9402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32407ee797ba4359a5f79513e69741ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.215855 9408 tablet_service.cc:1467] Processing CreateTablet for tablet 3db2ebff0e144fc1a1a5144e9c6f7e60 (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240426 09:54:12.216987 9408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db2ebff0e144fc1a1a5144e9c6f7e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.219718 9401 tablet_service.cc:1467] Processing CreateTablet for tablet ff15116e7e7546b1b7b04c5b227e503e (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240426 09:54:12.219861 9403 tablet_service.cc:1467] Processing CreateTablet for tablet 313a150ad860483a978ae5584f5554e0 (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240426 09:54:12.220891 9401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff15116e7e7546b1b7b04c5b227e503e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.221586 9404 tablet_service.cc:1467] Processing CreateTablet for tablet 75da9d0d79d24604a4bb97685008c10e (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240426 09:54:12.223934 9406 tablet_service.cc:1467] Processing CreateTablet for tablet 9bbd3b59d2fc41d49d30fc63daf3d20c (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240426 09:54:12.225775 9405 tablet_service.cc:1467] Processing CreateTablet for tablet b1990a88d033491bbf9904c47e91a23d (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240426 09:54:12.226691 9553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75da9d0d79d24604a4bb97685008c10e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.227111 9404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75da9d0d79d24604a4bb97685008c10e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.229027 9406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bbd3b59d2fc41d49d30fc63daf3d20c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.230840 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 87f66b3dfc3d4b5abb5726efb3d225f1 (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240426 09:54:12.231310 9405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1990a88d033491bbf9904c47e91a23d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.232020 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f66b3dfc3d4b5abb5726efb3d225f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.233144 9479 tablet_service.cc:1467] Processing CreateTablet for tablet 75da9d0d79d24604a4bb97685008c10e (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240426 09:54:12.234342 9479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75da9d0d79d24604a4bb97685008c10e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.235361 9403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313a150ad860483a978ae5584f5554e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.235881 9483 tablet_service.cc:1467] Processing CreateTablet for tablet 3db2ebff0e144fc1a1a5144e9c6f7e60 (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240426 09:54:12.236135 9478 tablet_service.cc:1467] Processing CreateTablet for tablet 313a150ad860483a978ae5584f5554e0 (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240426 09:54:12.237090 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db2ebff0e144fc1a1a5144e9c6f7e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.238647 9477 tablet_service.cc:1467] Processing CreateTablet for tablet 32407ee797ba4359a5f79513e69741ad (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240426 09:54:12.239888 9477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32407ee797ba4359a5f79513e69741ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.239718 9482 tablet_service.cc:1467] Processing CreateTablet for tablet b488843f1cb64132982e15fc409939fb (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240426 09:54:12.240885 9482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b488843f1cb64132982e15fc409939fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.241421 9478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313a150ad860483a978ae5584f5554e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.242825 9481 tablet_service.cc:1467] Processing CreateTablet for tablet 9bbd3b59d2fc41d49d30fc63daf3d20c (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240426 09:54:12.244007 9481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bbd3b59d2fc41d49d30fc63daf3d20c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.252861 9480 tablet_service.cc:1467] Processing CreateTablet for tablet b1990a88d033491bbf9904c47e91a23d (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240426 09:54:12.253306 9476 tablet_service.cc:1467] Processing CreateTablet for tablet ff15116e7e7546b1b7b04c5b227e503e (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240426 09:54:12.255290 9480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1990a88d033491bbf9904c47e91a23d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.256093 9484 tablet_service.cc:1467] Processing CreateTablet for tablet 99873d472db84e08889ea87be1b4d6df (DEFAULT_TABLE table=test-workload [id=f2c26a5fcf56478a9a898b8cca612f59]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240426 09:54:12.257247 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99873d472db84e08889ea87be1b4d6df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.263056 9476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff15116e7e7546b1b7b04c5b227e503e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:12.283885 9609 tablet_bootstrap.cc:492] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Bootstrap starting.
I20240426 09:54:12.333688 9611 tablet_bootstrap.cc:492] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa: Bootstrap starting.
I20240426 09:54:12.332849 9610 tablet_bootstrap.cc:492] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653: Bootstrap starting.
I20240426 09:54:12.343932 9610 tablet_bootstrap.cc:654] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.345408 9611 tablet_bootstrap.cc:654] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.382685 9610 tablet_bootstrap.cc:492] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653: No bootstrap required, opened a new log
I20240426 09:54:12.383162 9610 ts_tablet_manager.cc:1367] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653: Time spent bootstrapping tablet: real 0.051s user 0.009s sys 0.004s
I20240426 09:54:12.384042 9611 tablet_bootstrap.cc:492] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa: No bootstrap required, opened a new log
I20240426 09:54:12.384434 9611 ts_tablet_manager.cc:1367] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa: Time spent bootstrapping tablet: real 0.051s user 0.014s sys 0.015s
I20240426 09:54:12.385673 9610 raft_consensus.cc:348] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.386415 9610 raft_consensus.cc:374] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.386788 9611 raft_consensus.cc:348] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.387555 9611 raft_consensus.cc:374] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.387836 9611 raft_consensus.cc:729] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.388604 9611 consensus_queue.cc:260] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.392544 9609 tablet_bootstrap.cc:654] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.386703 9610 raft_consensus.cc:729] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b95908620241b7b6992d56d159a653, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.407598 9610 consensus_queue.cc:260] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.409750 9520 heartbeater.cc:503] Master 127.8.141.190:35665 was elected leader, sending a full tablet report...
I20240426 09:54:12.410248 9610 ts_tablet_manager.cc:1398] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653: Time spent starting tablet: real 0.027s user 0.006s sys 0.001s
I20240426 09:54:12.411293 9610 tablet_bootstrap.cc:492] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653: Bootstrap starting.
I20240426 09:54:12.411329 9445 heartbeater.cc:503] Master 127.8.141.190:35665 was elected leader, sending a full tablet report...
I20240426 09:54:12.412446 9611 ts_tablet_manager.cc:1398] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa: Time spent starting tablet: real 0.028s user 0.016s sys 0.008s
I20240426 09:54:12.413333 9611 tablet_bootstrap.cc:492] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa: Bootstrap starting.
I20240426 09:54:12.415904 9610 tablet_bootstrap.cc:654] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.418689 9611 tablet_bootstrap.cc:654] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.420917 9609 tablet_bootstrap.cc:492] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: No bootstrap required, opened a new log
I20240426 09:54:12.421377 9609 ts_tablet_manager.cc:1367] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent bootstrapping tablet: real 0.138s user 0.007s sys 0.006s
I20240426 09:54:12.423853 9609 raft_consensus.cc:348] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } }
I20240426 09:54:12.424629 9609 raft_consensus.cc:374] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.424916 9609 raft_consensus.cc:729] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.425662 9609 consensus_queue.cc:260] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } }
I20240426 09:54:12.465632 9594 heartbeater.cc:503] Master 127.8.141.190:35665 was elected leader, sending a full tablet report...
I20240426 09:54:12.466322 9611 tablet_bootstrap.cc:492] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa: No bootstrap required, opened a new log
I20240426 09:54:12.466717 9610 tablet_bootstrap.cc:492] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653: No bootstrap required, opened a new log
I20240426 09:54:12.466892 9611 ts_tablet_manager.cc:1367] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa: Time spent bootstrapping tablet: real 0.054s user 0.017s sys 0.034s
I20240426 09:54:12.466909 9609 ts_tablet_manager.cc:1398] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent starting tablet: real 0.045s user 0.031s sys 0.011s
I20240426 09:54:12.467198 9610 ts_tablet_manager.cc:1367] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653: Time spent bootstrapping tablet: real 0.056s user 0.024s sys 0.031s
I20240426 09:54:12.468214 9609 tablet_bootstrap.cc:492] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Bootstrap starting.
I20240426 09:54:12.469827 9611 raft_consensus.cc:348] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.470080 9610 raft_consensus.cc:348] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.470624 9611 raft_consensus.cc:374] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.470902 9610 raft_consensus.cc:374] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.471028 9611 raft_consensus.cc:729] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.471305 9610 raft_consensus.cc:729] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b95908620241b7b6992d56d159a653, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.471848 9611 consensus_queue.cc:260] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.472257 9610 consensus_queue.cc:260] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.474052 9611 ts_tablet_manager.cc:1398] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240426 09:54:12.474885 9609 tablet_bootstrap.cc:654] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.475057 9611 tablet_bootstrap.cc:492] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa: Bootstrap starting.
I20240426 09:54:12.475988 9610 ts_tablet_manager.cc:1398] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240426 09:54:12.476851 9610 tablet_bootstrap.cc:492] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653: Bootstrap starting.
I20240426 09:54:12.481365 9611 tablet_bootstrap.cc:654] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.482148 9610 tablet_bootstrap.cc:654] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.486394 9609 tablet_bootstrap.cc:492] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19: No bootstrap required, opened a new log
I20240426 09:54:12.486934 9609 ts_tablet_manager.cc:1367] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.012s
I20240426 09:54:12.489622 9609 raft_consensus.cc:348] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.490496 9609 raft_consensus.cc:374] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.490861 9609 raft_consensus.cc:729] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.491667 9609 consensus_queue.cc:260] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
W20240426 09:54:12.492636 9521 tablet.cc:2369] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 09:54:12.492766 9611 tablet_bootstrap.cc:492] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa: No bootstrap required, opened a new log
I20240426 09:54:12.493273 9611 ts_tablet_manager.cc:1367] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20240426 09:54:12.495579 9609 ts_tablet_manager.cc:1398] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20240426 09:54:12.495625 9610 tablet_bootstrap.cc:492] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653: No bootstrap required, opened a new log
I20240426 09:54:12.495695 9611 raft_consensus.cc:348] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.496340 9610 ts_tablet_manager.cc:1367] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.003s
I20240426 09:54:12.496562 9609 tablet_bootstrap.cc:492] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Bootstrap starting.
I20240426 09:54:12.496595 9611 raft_consensus.cc:374] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.497107 9611 raft_consensus.cc:729] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.497895 9611 consensus_queue.cc:260] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.498910 9610 raft_consensus.cc:348] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.499665 9610 raft_consensus.cc:374] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.499986 9610 raft_consensus.cc:729] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b95908620241b7b6992d56d159a653, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.500742 9610 consensus_queue.cc:260] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.501533 9611 ts_tablet_manager.cc:1398] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240426 09:54:12.502426 9609 tablet_bootstrap.cc:654] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.502472 9611 tablet_bootstrap.cc:492] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa: Bootstrap starting.
I20240426 09:54:12.504892 9610 ts_tablet_manager.cc:1398] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240426 09:54:12.505791 9610 tablet_bootstrap.cc:492] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653: Bootstrap starting.
I20240426 09:54:12.508500 9611 tablet_bootstrap.cc:654] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.511037 9610 tablet_bootstrap.cc:654] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.513835 9609 tablet_bootstrap.cc:492] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19: No bootstrap required, opened a new log
I20240426 09:54:12.514340 9609 ts_tablet_manager.cc:1367] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.007s
I20240426 09:54:12.516942 9609 raft_consensus.cc:348] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.517787 9609 raft_consensus.cc:374] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.518133 9609 raft_consensus.cc:729] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.519032 9609 consensus_queue.cc:260] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.520350 9611 tablet_bootstrap.cc:492] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa: No bootstrap required, opened a new log
I20240426 09:54:12.520802 9611 ts_tablet_manager.cc:1367] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.005s
I20240426 09:54:12.521562 9609 ts_tablet_manager.cc:1398] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20240426 09:54:12.522460 9609 tablet_bootstrap.cc:492] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Bootstrap starting.
I20240426 09:54:12.523252 9611 raft_consensus.cc:348] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.524016 9611 raft_consensus.cc:374] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.524361 9611 raft_consensus.cc:729] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.525125 9611 consensus_queue.cc:260] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.528100 9609 tablet_bootstrap.cc:654] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.528874 9611 ts_tablet_manager.cc:1398] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240426 09:54:12.530052 9611 tablet_bootstrap.cc:492] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa: Bootstrap starting.
I20240426 09:54:12.530618 9610 tablet_bootstrap.cc:492] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653: No bootstrap required, opened a new log
I20240426 09:54:12.531127 9610 ts_tablet_manager.cc:1367] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.011s
I20240426 09:54:12.533668 9610 raft_consensus.cc:348] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.534404 9610 raft_consensus.cc:374] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.534754 9610 raft_consensus.cc:729] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b95908620241b7b6992d56d159a653, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.535606 9611 tablet_bootstrap.cc:654] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.535547 9610 consensus_queue.cc:260] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.539050 9615 raft_consensus.cc:482] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:12.539455 9615 raft_consensus.cc:504] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.540046 9609 tablet_bootstrap.cc:492] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: No bootstrap required, opened a new log
I20240426 09:54:12.540645 9609 ts_tablet_manager.cc:1367] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.003s
I20240426 09:54:12.541709 9615 leader_election.cc:288] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.543383 9609 raft_consensus.cc:348] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.544319 9609 raft_consensus.cc:374] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.544661 9609 raft_consensus.cc:729] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.545572 9614 raft_consensus.cc:482] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:12.545432 9609 consensus_queue.cc:260] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.546131 9614 raft_consensus.cc:504] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.546993 9611 tablet_bootstrap.cc:492] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa: No bootstrap required, opened a new log
I20240426 09:54:12.547410 9611 ts_tablet_manager.cc:1367] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.005s
I20240426 09:54:12.547938 9609 ts_tablet_manager.cc:1398] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20240426 09:54:12.548565 9614 leader_election.cc:288] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.548808 9609 tablet_bootstrap.cc:492] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Bootstrap starting.
I20240426 09:54:12.549854 9611 raft_consensus.cc:348] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.550560 9610 ts_tablet_manager.cc:1398] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20240426 09:54:12.552040 9610 tablet_bootstrap.cc:492] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653: Bootstrap starting.
I20240426 09:54:12.550683 9611 raft_consensus.cc:374] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.553432 9611 raft_consensus.cc:729] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.554212 9609 tablet_bootstrap.cc:654] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.554134 9611 consensus_queue.cc:260] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.556614 9611 ts_tablet_manager.cc:1398] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240426 09:54:12.557564 9611 tablet_bootstrap.cc:492] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa: Bootstrap starting.
I20240426 09:54:12.562453 9610 tablet_bootstrap.cc:654] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.562927 9611 tablet_bootstrap.cc:654] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.584653 9611 tablet_bootstrap.cc:492] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa: No bootstrap required, opened a new log
I20240426 09:54:12.586346 9611 ts_tablet_manager.cc:1367] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.000s
I20240426 09:54:12.594746 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313a150ad860483a978ae5584f5554e0" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b95908620241b7b6992d56d159a653" is_pre_election: true
W20240426 09:54:12.596186 9383 leader_election.cc:341] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:12.590384 9610 tablet_bootstrap.cc:492] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653: No bootstrap required, opened a new log
I20240426 09:54:12.590503 9420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db2ebff0e144fc1a1a5144e9c6f7e60" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa" is_pre_election: true
I20240426 09:54:12.597107 9420 raft_consensus.cc:2454] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 0.
I20240426 09:54:12.597040 9611 raft_consensus.cc:348] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.597939 9611 raft_consensus.cc:374] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.598245 9609 tablet_bootstrap.cc:492] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19: No bootstrap required, opened a new log
I20240426 09:54:12.598492 9461 leader_election.cc:302] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:12.598729 9609 ts_tablet_manager.cc:1367] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent bootstrapping tablet: real 0.050s user 0.021s sys 0.025s
I20240426 09:54:12.598249 9611 raft_consensus.cc:729] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.599321 9615 raft_consensus.cc:2781] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:12.599684 9615 raft_consensus.cc:482] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:12.600075 9615 raft_consensus.cc:3037] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.600507 9569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313a150ad860483a978ae5584f5554e0" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" is_pre_election: true
I20240426 09:54:12.600073 9611 consensus_queue.cc:260] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.601759 9610 ts_tablet_manager.cc:1367] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.000s
I20240426 09:54:12.601704 9609 raft_consensus.cc:348] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.602463 9609 raft_consensus.cc:374] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.602746 9609 raft_consensus.cc:729] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.603408 9609 consensus_queue.cc:260] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
W20240426 09:54:12.603824 9386 leader_election.cc:341] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:12.604269 9386 leader_election.cc:302] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a51b16892b1f45a6933030491fcc05aa; no voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653
I20240426 09:54:12.605482 9609 ts_tablet_manager.cc:1398] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20240426 09:54:12.606328 9609 tablet_bootstrap.cc:492] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Bootstrap starting.
I20240426 09:54:12.606693 9615 raft_consensus.cc:504] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.607919 9610 raft_consensus.cc:348] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.608490 9615 leader_election.cc:288] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 election: Requested vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.608675 9610 raft_consensus.cc:374] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.609005 9610 raft_consensus.cc:729] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b95908620241b7b6992d56d159a653, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.609817 9420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db2ebff0e144fc1a1a5144e9c6f7e60" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
I20240426 09:54:12.609820 9610 consensus_queue.cc:260] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.610530 9420 raft_consensus.cc:3037] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.611372 9619 raft_consensus.cc:482] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:12.611485 9609 tablet_bootstrap.cc:654] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.612006 9610 ts_tablet_manager.cc:1398] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240426 09:54:12.611856 9619 raft_consensus.cc:504] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } }
I20240426 09:54:12.614573 9610 tablet_bootstrap.cc:492] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653: Bootstrap starting.
I20240426 09:54:12.615962 9619 leader_election.cc:288] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215)
I20240426 09:54:12.616472 9614 raft_consensus.cc:2726] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:12.616598 9420 raft_consensus.cc:2454] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 1.
I20240426 09:54:12.623600 9461 leader_election.cc:302] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:12.624460 9615 raft_consensus.cc:2781] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:12.627442 9615 raft_consensus.cc:686] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Becoming Leader. State: Replica: 88b95908620241b7b6992d56d159a653, State: Running, Role: LEADER
I20240426 09:54:12.629788 9568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db2ebff0e144fc1a1a5144e9c6f7e60" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:12.629812 9614 raft_consensus.cc:482] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:12.630450 9568 raft_consensus.cc:3037] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.630650 9614 raft_consensus.cc:504] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.629611 9611 ts_tablet_manager.cc:1398] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20240426 09:54:12.628376 9615 consensus_queue.cc:237] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.632535 9614 leader_election.cc:288] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.629650 9569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db2ebff0e144fc1a1a5144e9c6f7e60" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" is_pre_election: true
I20240426 09:54:12.633206 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32407ee797ba4359a5f79513e69741ad" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b95908620241b7b6992d56d159a653" is_pre_election: true
I20240426 09:54:12.633750 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32407ee797ba4359a5f79513e69741ad" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" is_pre_election: true
W20240426 09:54:12.636390 9383 leader_election.cc:341] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:12.636893 9568 raft_consensus.cc:2454] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 1.
W20240426 09:54:12.638305 9386 leader_election.cc:341] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:12.638725 9386 leader_election.cc:302] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a51b16892b1f45a6933030491fcc05aa; no voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653
I20240426 09:54:12.639824 9610 tablet_bootstrap.cc:654] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.640805 9614 raft_consensus.cc:2726] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:12.642531 9609 tablet_bootstrap.cc:492] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19: No bootstrap required, opened a new log
I20240426 09:54:12.643095 9609 ts_tablet_manager.cc:1367] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent bootstrapping tablet: real 0.037s user 0.018s sys 0.003s
I20240426 09:54:12.643381 9611 tablet_bootstrap.cc:492] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa: Bootstrap starting.
I20240426 09:54:12.645879 9609 raft_consensus.cc:348] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.649344 9609 raft_consensus.cc:374] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.646803 9639 raft_consensus.cc:482] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:12.649667 9619 raft_consensus.cc:482] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:12.650100 9639 raft_consensus.cc:504] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.650185 9619 raft_consensus.cc:504] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.652051 9639 leader_election.cc:288] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.649706 9609 raft_consensus.cc:729] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.652266 9619 leader_election.cc:288] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215)
I20240426 09:54:12.652637 9420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa" is_pre_election: true
I20240426 09:54:12.653256 9609 consensus_queue.cc:260] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.653638 9568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" is_pre_election: true
W20240426 09:54:12.653839 9461 leader_election.cc:341] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:12.654366 9568 raft_consensus.cc:2454] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 0.
I20240426 09:54:12.654417 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f66b3dfc3d4b5abb5726efb3d225f1" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b95908620241b7b6992d56d159a653" is_pre_election: true
I20240426 09:54:12.655402 9461 leader_election.cc:302] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653; no voters: a51b16892b1f45a6933030491fcc05aa
I20240426 09:54:12.656128 9639 raft_consensus.cc:2781] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:12.656287 9609 ts_tablet_manager.cc:1398] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20240426 09:54:12.656575 9639 raft_consensus.cc:482] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:12.656962 9639 raft_consensus.cc:3037] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.657161 9609 tablet_bootstrap.cc:492] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Bootstrap starting.
I20240426 09:54:12.659742 9614 raft_consensus.cc:482] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:12.660179 9614 raft_consensus.cc:504] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.661937 9614 leader_election.cc:288] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.662864 9609 tablet_bootstrap.cc:654] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.663295 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99873d472db84e08889ea87be1b4d6df" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b95908620241b7b6992d56d159a653" is_pre_election: true
I20240426 09:54:12.663748 9568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99873d472db84e08889ea87be1b4d6df" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" is_pre_election: true
I20240426 09:54:12.664413 9568 raft_consensus.cc:2454] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 0.
I20240426 09:54:12.665413 9386 leader_election.cc:302] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:12.663863 9495 raft_consensus.cc:2454] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 0.
I20240426 09:54:12.667116 9610 tablet_bootstrap.cc:492] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653: No bootstrap required, opened a new log
I20240426 09:54:12.667562 9610 ts_tablet_manager.cc:1367] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653: Time spent bootstrapping tablet: real 0.053s user 0.010s sys 0.000s
I20240426 09:54:12.667502 9614 raft_consensus.cc:2781] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:12.668104 9614 raft_consensus.cc:482] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:12.668483 9614 raft_consensus.cc:3037] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.670322 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99873d472db84e08889ea87be1b4d6df" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b95908620241b7b6992d56d159a653" is_pre_election: true
I20240426 09:54:12.670173 9610 raft_consensus.cc:348] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.670897 9610 raft_consensus.cc:374] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240426 09:54:12.671203 9532 leader_election.cc:341] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:12.670918 9495 raft_consensus.cc:2454] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in term 0.
I20240426 09:54:12.671209 9610 raft_consensus.cc:729] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b95908620241b7b6992d56d159a653, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.672850 9610 consensus_queue.cc:260] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.674127 9532 leader_election.cc:302] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653; no voters:
I20240426 09:54:12.674880 9619 raft_consensus.cc:2781] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:12.675375 9619 raft_consensus.cc:482] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:12.675797 9619 raft_consensus.cc:3037] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.682519 9619 raft_consensus.cc:504] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.684496 9614 raft_consensus.cc:504] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.685585 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99873d472db84e08889ea87be1b4d6df" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:12.688335 9495 raft_consensus.cc:3037] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.691186 9611 tablet_bootstrap.cc:654] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.691582 9615 raft_consensus.cc:482] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:12.692047 9615 raft_consensus.cc:504] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.693826 9615 leader_election.cc:288] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.695238 9495 raft_consensus.cc:2454] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in term 1.
I20240426 09:54:12.686106 9619 leader_election.cc:288] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 election: Requested vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215)
I20240426 09:54:12.697961 9532 leader_election.cc:302] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653; no voters:
I20240426 09:54:12.698714 9619 raft_consensus.cc:2781] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:12.701139 9609 tablet_bootstrap.cc:492] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19: No bootstrap required, opened a new log
I20240426 09:54:12.701617 9609 ts_tablet_manager.cc:1367] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent bootstrapping tablet: real 0.045s user 0.036s sys 0.004s
I20240426 09:54:12.701555 9568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99873d472db84e08889ea87be1b4d6df" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:12.702037 9420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f66b3dfc3d4b5abb5726efb3d225f1" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa" is_pre_election: true
I20240426 09:54:12.702535 9614 leader_election.cc:288] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 election: Requested vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.703351 9419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99873d472db84e08889ea87be1b4d6df" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa" is_pre_election: true
I20240426 09:54:12.698993 9639 raft_consensus.cc:504] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.704193 9419 raft_consensus.cc:2379] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in current term 1: Already voted for candidate a51b16892b1f45a6933030491fcc05aa in this term.
I20240426 09:54:12.704509 9619 raft_consensus.cc:686] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 LEADER]: Becoming Leader. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Running, Role: LEADER
I20240426 09:54:12.705077 9639 leader_election.cc:288] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 election: Requested vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.705189 9418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99873d472db84e08889ea87be1b4d6df" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
I20240426 09:54:12.705421 9619 consensus_queue.cc:237] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
W20240426 09:54:12.707155 9535 leader_election.cc:341] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:12.707629 9535 leader_election.cc:302] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19; no voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa
I20240426 09:54:12.708371 9609 raft_consensus.cc:348] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.706070 9418 raft_consensus.cc:2379] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in current term 1: Already voted for candidate a51b16892b1f45a6933030491fcc05aa in this term.
I20240426 09:54:12.709893 9609 raft_consensus.cc:374] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.710278 9609 raft_consensus.cc:729] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.711016 9609 consensus_queue.cc:260] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.711762 9646 raft_consensus.cc:2726] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:12.715518 9611 tablet_bootstrap.cc:492] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa: No bootstrap required, opened a new log
I20240426 09:54:12.715957 9611 ts_tablet_manager.cc:1367] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa: Time spent bootstrapping tablet: real 0.073s user 0.010s sys 0.000s
I20240426 09:54:12.717012 9610 ts_tablet_manager.cc:1398] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653: Time spent starting tablet: real 0.049s user 0.020s sys 0.009s
I20240426 09:54:12.717460 9418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313a150ad860483a978ae5584f5554e0" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa" is_pre_election: true
I20240426 09:54:12.718106 9610 tablet_bootstrap.cc:492] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653: Bootstrap starting.
I20240426 09:54:12.718132 9418 raft_consensus.cc:2454] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 0.
I20240426 09:54:12.720250 9568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313a150ad860483a978ae5584f5554e0" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" is_pre_election: true
I20240426 09:54:12.721333 9318 catalog_manager.cc:5484] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131). New cstate: current_term: 1 leader_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:12.722245 9418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
I20240426 09:54:12.723069 9619 raft_consensus.cc:482] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:12.723491 9619 raft_consensus.cc:504] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.724125 9610 tablet_bootstrap.cc:654] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.724215 9568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:12.724831 9568 raft_consensus.cc:3037] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.725329 9619 leader_election.cc:288] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215)
I20240426 09:54:12.726892 9418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1990a88d033491bbf9904c47e91a23d" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa" is_pre_election: true
I20240426 09:54:12.727303 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1990a88d033491bbf9904c47e91a23d" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b95908620241b7b6992d56d159a653" is_pre_election: true
I20240426 09:54:12.716324 9315 catalog_manager.cc:5484] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b95908620241b7b6992d56d159a653 (127.8.141.130). New cstate: current_term: 1 leader_uuid: "88b95908620241b7b6992d56d159a653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
W20240426 09:54:12.728621 9532 leader_election.cc:341] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215): Illegal state: must be running to vote when last-logged opid is not known
W20240426 09:54:12.729609 9535 leader_election.cc:341] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:12.730062 9535 leader_election.cc:302] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19; no voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa
I20240426 09:54:12.731091 9644 raft_consensus.cc:482] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:12.731330 9619 raft_consensus.cc:2726] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:12.731571 9644 raft_consensus.cc:504] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.734413 9644 leader_election.cc:288] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.735625 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99873d472db84e08889ea87be1b4d6df" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:12.736462 9495 raft_consensus.cc:2379] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a51b16892b1f45a6933030491fcc05aa in current term 1: Already voted for candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in this term.
W20240426 09:54:12.736707 9461 leader_election.cc:341] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:12.737123 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b488843f1cb64132982e15fc409939fb" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" is_pre_election: true
I20240426 09:54:12.737416 9461 leader_election.cc:302] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa; no voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19
I20240426 09:54:12.737684 9418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b488843f1cb64132982e15fc409939fb" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa" is_pre_election: true
I20240426 09:54:12.738220 9609 ts_tablet_manager.cc:1398] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent starting tablet: real 0.036s user 0.015s sys 0.012s
I20240426 09:54:12.738611 9567 raft_consensus.cc:2454] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 0.
I20240426 09:54:12.738889 9644 raft_consensus.cc:2781] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:12.739377 9609 tablet_bootstrap.cc:492] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Bootstrap starting.
I20240426 09:54:12.738273 9418 raft_consensus.cc:2454] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 0.
W20240426 09:54:12.738529 9461 leader_election.cc:341] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:12.739288 9644 raft_consensus.cc:482] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:12.734570 9611 raft_consensus.cc:348] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.741194 9611 raft_consensus.cc:374] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.741413 9611 raft_consensus.cc:729] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Initialized, Role: FOLLOWER
W20240426 09:54:12.741914 9366 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 09:54:12.741963 9611 consensus_queue.cc:260] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.742347 9644 raft_consensus.cc:3037] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.746029 9609 tablet_bootstrap.cc:654] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.746292 9611 ts_tablet_manager.cc:1398] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20240426 09:54:12.747212 9611 tablet_bootstrap.cc:492] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa: Bootstrap starting.
I20240426 09:54:12.748389 9568 raft_consensus.cc:2454] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 1.
I20240426 09:54:12.748994 9644 raft_consensus.cc:504] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.749657 9383 leader_election.cc:302] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a51b16892b1f45a6933030491fcc05aa; no voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653
I20240426 09:54:12.750604 9614 raft_consensus.cc:482] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:12.752305 9614 raft_consensus.cc:504] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.761261 9461 leader_election.cc:302] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653; no voters:
I20240426 09:54:12.762136 9643 raft_consensus.cc:2781] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:12.762162 9461 leader_election.cc:302] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653; no voters: a51b16892b1f45a6933030491fcc05aa
I20240426 09:54:12.762591 9643 raft_consensus.cc:482] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:12.762962 9643 raft_consensus.cc:3037] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.764715 9614 leader_election.cc:288] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.765190 9614 raft_consensus.cc:2726] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240426 09:54:12.765478 9639 raft_consensus.cc:2781] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:12.752530 9644 leader_election.cc:288] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 election: Requested vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.765990 9639 raft_consensus.cc:686] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Becoming Leader. State: Replica: 88b95908620241b7b6992d56d159a653, State: Running, Role: LEADER
I20240426 09:54:12.770298 9643 raft_consensus.cc:504] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.770543 9568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" is_pre_election: true
I20240426 09:54:12.771764 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313a150ad860483a978ae5584f5554e0" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:12.771845 9568 raft_consensus.cc:2454] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 0.
W20240426 09:54:12.773140 9461 leader_election.cc:341] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:12.774578 9386 leader_election.cc:302] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:12.774869 9639 consensus_queue.cc:237] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.775614 9643 leader_election.cc:288] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 election: Requested vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.775674 9614 raft_consensus.cc:2781] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:12.776228 9614 raft_consensus.cc:482] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:12.776615 9614 raft_consensus.cc:3037] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.780040 9568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b488843f1cb64132982e15fc409939fb" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:12.780243 9609 tablet_bootstrap.cc:492] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: No bootstrap required, opened a new log
I20240426 09:54:12.780606 9568 raft_consensus.cc:3037] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.780736 9609 ts_tablet_manager.cc:1367] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.010s
I20240426 09:54:12.782946 9609 raft_consensus.cc:348] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.783358 9609 raft_consensus.cc:374] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.783532 9609 raft_consensus.cc:729] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.783823 9418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313a150ad860483a978ae5584f5554e0" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
I20240426 09:54:12.783907 9609 consensus_queue.cc:260] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.784466 9418 raft_consensus.cc:3037] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.785348 9609 ts_tablet_manager.cc:1398] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240426 09:54:12.785914 9609 tablet_bootstrap.cc:492] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Bootstrap starting.
I20240426 09:54:12.786141 9614 raft_consensus.cc:504] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.787376 9419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b488843f1cb64132982e15fc409939fb" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
I20240426 09:54:12.787921 9419 raft_consensus.cc:3037] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.790876 9609 tablet_bootstrap.cc:654] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.791986 9418 raft_consensus.cc:2454] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 1.
I20240426 09:54:12.793761 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b95908620241b7b6992d56d159a653" is_pre_election: true
I20240426 09:54:12.793538 9461 leader_election.cc:302] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa; no voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19
I20240426 09:54:12.795658 9639 raft_consensus.cc:2781] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:12.795699 9419 raft_consensus.cc:2454] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 1.
I20240426 09:54:12.793844 9568 raft_consensus.cc:2454] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 1.
I20240426 09:54:12.796418 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:12.796211 9639 raft_consensus.cc:686] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Becoming Leader. State: Replica: 88b95908620241b7b6992d56d159a653, State: Running, Role: LEADER
I20240426 09:54:12.796628 9611 tablet_bootstrap.cc:654] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.798523 9610 tablet_bootstrap.cc:492] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653: No bootstrap required, opened a new log
I20240426 09:54:12.798218 9567 raft_consensus.cc:3037] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.798115 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:12.801448 9648 raft_consensus.cc:482] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:12.811793 9610 ts_tablet_manager.cc:1367] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653: Time spent bootstrapping tablet: real 0.094s user 0.019s sys 0.017s
I20240426 09:54:12.812465 9648 raft_consensus.cc:504] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.799435 9639 consensus_queue.cc:237] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.814733 9610 raft_consensus.cc:348] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.815446 9610 raft_consensus.cc:374] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.815727 9610 raft_consensus.cc:729] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b95908620241b7b6992d56d159a653, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.816334 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313a150ad860483a978ae5584f5554e0" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b95908620241b7b6992d56d159a653" is_pre_election: true
W20240426 09:54:12.799477 9383 leader_election.cc:341] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:12.817905 9648 leader_election.cc:288] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215)
I20240426 09:54:12.798506 9461 leader_election.cc:302] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:12.819010 9643 raft_consensus.cc:2781] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:12.819067 9419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313a150ad860483a978ae5584f5554e0" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa" is_pre_election: true
I20240426 09:54:12.819553 9643 raft_consensus.cc:686] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Becoming Leader. State: Replica: 88b95908620241b7b6992d56d159a653, State: Running, Role: LEADER
I20240426 09:54:12.819875 9419 raft_consensus.cc:2379] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in current term 1: Already voted for candidate 88b95908620241b7b6992d56d159a653 in this term.
I20240426 09:54:12.820415 9643 consensus_queue.cc:237] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.797685 9614 leader_election.cc:288] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 election: Requested vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.823244 9535 leader_election.cc:302] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19; no voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa
I20240426 09:54:12.826362 9648 raft_consensus.cc:3037] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Advancing to term 1
W20240426 09:54:12.828287 9383 leader_election.cc:341] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:12.825327 9567 raft_consensus.cc:2454] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 1.
I20240426 09:54:12.830826 9386 leader_election.cc:302] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, a51b16892b1f45a6933030491fcc05aa; no voters: 88b95908620241b7b6992d56d159a653
I20240426 09:54:12.831579 9614 raft_consensus.cc:2781] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:12.832010 9614 raft_consensus.cc:686] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 1 LEADER]: Becoming Leader. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:12.816363 9610 consensus_queue.cc:260] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.832784 9614 consensus_queue.cc:237] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.834802 9611 tablet_bootstrap.cc:492] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa: No bootstrap required, opened a new log
I20240426 09:54:12.835247 9611 ts_tablet_manager.cc:1367] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa: Time spent bootstrapping tablet: real 0.088s user 0.010s sys 0.001s
I20240426 09:54:12.837453 9648 raft_consensus.cc:2726] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:12.837507 9651 raft_consensus.cc:482] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:12.837745 9611 raft_consensus.cc:348] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } }
I20240426 09:54:12.838449 9611 raft_consensus.cc:374] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.838099 9651 raft_consensus.cc:504] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.838755 9611 raft_consensus.cc:729] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.839397 9611 consensus_queue.cc:260] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } }
I20240426 09:54:12.840112 9610 ts_tablet_manager.cc:1398] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653: Time spent starting tablet: real 0.028s user 0.000s sys 0.006s
I20240426 09:54:12.840211 9651 leader_election.cc:288] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.842288 9610 tablet_bootstrap.cc:492] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653: Bootstrap starting.
I20240426 09:54:12.842761 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff15116e7e7546b1b7b04c5b227e503e" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b95908620241b7b6992d56d159a653" is_pre_election: true
I20240426 09:54:12.843374 9495 raft_consensus.cc:2454] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 0.
I20240426 09:54:12.838600 9315 catalog_manager.cc:5484] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b95908620241b7b6992d56d159a653 (127.8.141.130). New cstate: current_term: 1 leader_uuid: "88b95908620241b7b6992d56d159a653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:12.844178 9609 tablet_bootstrap.cc:492] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19: No bootstrap required, opened a new log
I20240426 09:54:12.844568 9383 leader_election.cc:302] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:12.845347 9651 raft_consensus.cc:2781] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:12.845728 9651 raft_consensus.cc:482] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:12.846046 9651 raft_consensus.cc:3037] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.848397 9610 tablet_bootstrap.cc:654] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.844599 9609 ts_tablet_manager.cc:1367] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent bootstrapping tablet: real 0.059s user 0.007s sys 0.017s
I20240426 09:54:12.853303 9611 ts_tablet_manager.cc:1398] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa: Time spent starting tablet: real 0.018s user 0.010s sys 0.000s
I20240426 09:54:12.853214 9609 raft_consensus.cc:348] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.853953 9609 raft_consensus.cc:374] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.854223 9611 tablet_bootstrap.cc:492] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa: Bootstrap starting.
I20240426 09:54:12.844966 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff15116e7e7546b1b7b04c5b227e503e" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" is_pre_election: true
I20240426 09:54:12.853646 9318 catalog_manager.cc:5484] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa reported cstate change: term changed from 0 to 1, leader changed from <none> to a51b16892b1f45a6933030491fcc05aa (127.8.141.129). New cstate: current_term: 1 leader_uuid: "a51b16892b1f45a6933030491fcc05aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:12.854230 9609 raft_consensus.cc:729] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.844449 9315 catalog_manager.cc:5484] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b95908620241b7b6992d56d159a653 (127.8.141.130). New cstate: current_term: 1 leader_uuid: "88b95908620241b7b6992d56d159a653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:12.857451 9610 tablet_bootstrap.cc:492] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653: No bootstrap required, opened a new log
I20240426 09:54:12.856712 9315 catalog_manager.cc:5484] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b95908620241b7b6992d56d159a653 (127.8.141.130). New cstate: current_term: 1 leader_uuid: "88b95908620241b7b6992d56d159a653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:12.857872 9609 consensus_queue.cc:260] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.857890 9610 ts_tablet_manager.cc:1367] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20240426 09:54:12.859107 9567 raft_consensus.cc:2454] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 0.
I20240426 09:54:12.860319 9609 ts_tablet_manager.cc:1398] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240426 09:54:12.863620 9610 raft_consensus.cc:348] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.864177 9609 tablet_bootstrap.cc:492] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Bootstrap starting.
I20240426 09:54:12.864305 9610 raft_consensus.cc:374] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.864573 9610 raft_consensus.cc:729] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b95908620241b7b6992d56d159a653, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.865142 9610 consensus_queue.cc:260] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.867009 9610 ts_tablet_manager.cc:1398] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240426 09:54:12.867760 9610 tablet_bootstrap.cc:492] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653: Bootstrap starting.
I20240426 09:54:12.869166 9609 tablet_bootstrap.cc:654] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.861053 9651 raft_consensus.cc:504] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.859207 9611 tablet_bootstrap.cc:654] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.873239 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff15116e7e7546b1b7b04c5b227e503e" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:12.873886 9610 tablet_bootstrap.cc:654] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.873788 9495 raft_consensus.cc:3037] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.875964 9651 leader_election.cc:288] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 election: Requested vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.877080 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff15116e7e7546b1b7b04c5b227e503e" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:12.879261 9567 raft_consensus.cc:3037] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.882601 9495 raft_consensus.cc:2454] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 1.
I20240426 09:54:12.883641 9383 leader_election.cc:302] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:12.884418 9651 raft_consensus.cc:2781] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:12.884851 9651 raft_consensus.cc:686] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 1 LEADER]: Becoming Leader. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:12.885598 9651 consensus_queue.cc:237] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.887738 9567 raft_consensus.cc:2454] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 1.
I20240426 09:54:12.892064 9609 tablet_bootstrap.cc:492] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19: No bootstrap required, opened a new log
I20240426 09:54:12.892534 9609 ts_tablet_manager.cc:1367] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.003s
I20240426 09:54:12.892757 9610 tablet_bootstrap.cc:492] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653: No bootstrap required, opened a new log
I20240426 09:54:12.893169 9610 ts_tablet_manager.cc:1367] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.000s
I20240426 09:54:12.893568 9611 tablet_bootstrap.cc:492] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa: No bootstrap required, opened a new log
I20240426 09:54:12.894003 9611 ts_tablet_manager.cc:1367] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.008s
I20240426 09:54:12.894866 9609 raft_consensus.cc:348] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.895546 9609 raft_consensus.cc:374] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.895799 9609 raft_consensus.cc:729] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.896462 9611 raft_consensus.cc:348] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.896471 9609 consensus_queue.cc:260] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.897411 9611 raft_consensus.cc:374] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.897764 9611 raft_consensus.cc:729] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.897996 9610 raft_consensus.cc:348] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.898720 9610 raft_consensus.cc:374] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.898432 9611 consensus_queue.cc:260] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.899004 9610 raft_consensus.cc:729] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b95908620241b7b6992d56d159a653, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.899753 9610 consensus_queue.cc:260] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.900739 9611 ts_tablet_manager.cc:1398] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240426 09:54:12.901571 9611 tablet_bootstrap.cc:492] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa: Bootstrap starting.
I20240426 09:54:12.901765 9610 ts_tablet_manager.cc:1398] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240426 09:54:12.902527 9610 tablet_bootstrap.cc:492] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653: Bootstrap starting.
I20240426 09:54:12.902557 9318 catalog_manager.cc:5484] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa reported cstate change: term changed from 0 to 1, leader changed from <none> to a51b16892b1f45a6933030491fcc05aa (127.8.141.129). New cstate: current_term: 1 leader_uuid: "a51b16892b1f45a6933030491fcc05aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:12.905369 9609 ts_tablet_manager.cc:1398] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20240426 09:54:12.906602 9611 tablet_bootstrap.cc:654] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.907748 9610 tablet_bootstrap.cc:654] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:12.920608 9611 tablet_bootstrap.cc:492] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa: No bootstrap required, opened a new log
I20240426 09:54:12.921005 9611 ts_tablet_manager.cc:1367] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.003s
I20240426 09:54:12.922132 9651 raft_consensus.cc:482] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:12.922566 9651 raft_consensus.cc:504] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.923228 9610 tablet_bootstrap.cc:492] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653: No bootstrap required, opened a new log
I20240426 09:54:12.923620 9610 ts_tablet_manager.cc:1367] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.010s
I20240426 09:54:12.923485 9611 raft_consensus.cc:348] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.924384 9651 leader_election.cc:288] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.924154 9611 raft_consensus.cc:374] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.925141 9611 raft_consensus.cc:729] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.925036 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1990a88d033491bbf9904c47e91a23d" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b95908620241b7b6992d56d159a653" is_pre_election: true
I20240426 09:54:12.925303 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1990a88d033491bbf9904c47e91a23d" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" is_pre_election: true
I20240426 09:54:12.925730 9495 raft_consensus.cc:2454] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 0.
I20240426 09:54:12.925935 9567 raft_consensus.cc:2454] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 0.
I20240426 09:54:12.925854 9610 raft_consensus.cc:348] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } }
I20240426 09:54:12.925885 9611 consensus_queue.cc:260] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.926708 9610 raft_consensus.cc:374] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:12.926999 9610 raft_consensus.cc:729] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b95908620241b7b6992d56d159a653, State: Initialized, Role: FOLLOWER
I20240426 09:54:12.927156 9383 leader_election.cc:302] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:12.928011 9611 ts_tablet_manager.cc:1398] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240426 09:54:12.927667 9610 consensus_queue.cc:260] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } }
I20240426 09:54:12.928517 9661 raft_consensus.cc:2781] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:12.928829 9661 raft_consensus.cc:482] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:12.929144 9661 raft_consensus.cc:3037] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.929731 9610 ts_tablet_manager.cc:1398] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240426 09:54:12.936122 9661 raft_consensus.cc:504] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.937963 9661 leader_election.cc:288] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 election: Requested vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:12.938665 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1990a88d033491bbf9904c47e91a23d" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:12.939237 9495 raft_consensus.cc:3037] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.939373 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1990a88d033491bbf9904c47e91a23d" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:12.939952 9567 raft_consensus.cc:3037] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.943862 9495 raft_consensus.cc:2454] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 1.
I20240426 09:54:12.944942 9383 leader_election.cc:302] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:12.945374 9567 raft_consensus.cc:2454] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 1.
I20240426 09:54:12.945672 9661 raft_consensus.cc:2781] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:12.946123 9661 raft_consensus.cc:686] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 1 LEADER]: Becoming Leader. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:12.946750 9661 consensus_queue.cc:237] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:12.953392 9318 catalog_manager.cc:5484] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa reported cstate change: term changed from 0 to 1, leader changed from <none> to a51b16892b1f45a6933030491fcc05aa (127.8.141.129). New cstate: current_term: 1 leader_uuid: "a51b16892b1f45a6933030491fcc05aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:12.969859 9639 raft_consensus.cc:482] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:12.970242 9639 raft_consensus.cc:504] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } }
I20240426 09:54:12.971673 9639 leader_election.cc:288] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209), a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159)
I20240426 09:54:12.972280 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f66b3dfc3d4b5abb5726efb3d225f1" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" is_pre_election: true
I20240426 09:54:12.972517 9419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f66b3dfc3d4b5abb5726efb3d225f1" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa" is_pre_election: true
I20240426 09:54:12.972743 9567 raft_consensus.cc:2454] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 0.
I20240426 09:54:12.973040 9419 raft_consensus.cc:2454] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 0.
I20240426 09:54:12.973515 9461 leader_election.cc:302] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653; no voters:
I20240426 09:54:12.974069 9639 raft_consensus.cc:2781] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:12.974399 9639 raft_consensus.cc:482] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:12.974643 9639 raft_consensus.cc:3037] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.978515 9639 raft_consensus.cc:504] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } }
I20240426 09:54:12.979926 9639 leader_election.cc:288] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 election: Requested vote from peers 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209), a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159)
I20240426 09:54:12.980520 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f66b3dfc3d4b5abb5726efb3d225f1" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:12.980728 9419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f66b3dfc3d4b5abb5726efb3d225f1" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
I20240426 09:54:12.980954 9567 raft_consensus.cc:3037] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.981199 9419 raft_consensus.cc:3037] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:12.984869 9567 raft_consensus.cc:2454] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 1.
I20240426 09:54:12.985169 9419 raft_consensus.cc:2454] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 1.
I20240426 09:54:12.985651 9461 leader_election.cc:302] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653; no voters:
I20240426 09:54:12.986289 9639 raft_consensus.cc:2781] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:12.986619 9639 raft_consensus.cc:686] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Becoming Leader. State: Replica: 88b95908620241b7b6992d56d159a653, State: Running, Role: LEADER
I20240426 09:54:12.987339 9639 consensus_queue.cc:237] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } }
I20240426 09:54:12.993368 9318 catalog_manager.cc:5484] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b95908620241b7b6992d56d159a653 (127.8.141.130). New cstate: current_term: 1 leader_uuid: "88b95908620241b7b6992d56d159a653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:13.078037 9639 consensus_queue.cc:1035] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.088847 9643 consensus_queue.cc:1035] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.131410 9648 consensus_queue.cc:1035] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.140867 9648 raft_consensus.cc:482] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:13.141316 9648 raft_consensus.cc:504] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.144137 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32407ee797ba4359a5f79513e69741ad" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b95908620241b7b6992d56d159a653" is_pre_election: true
I20240426 09:54:13.144798 9495 raft_consensus.cc:2454] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in term 0.
I20240426 09:54:13.145205 9619 consensus_queue.cc:1035] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.146553 9532 leader_election.cc:302] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653; no voters:
I20240426 09:54:13.147353 9619 raft_consensus.cc:2781] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:13.147697 9619 raft_consensus.cc:482] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:13.148000 9619 raft_consensus.cc:3037] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:13.151913 9648 leader_election.cc:288] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215)
I20240426 09:54:13.153892 9419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32407ee797ba4359a5f79513e69741ad" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa" is_pre_election: true
I20240426 09:54:13.153965 9619 raft_consensus.cc:504] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.154371 9419 raft_consensus.cc:2454] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in term 0.
I20240426 09:54:13.156526 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32407ee797ba4359a5f79513e69741ad" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:13.157064 9495 raft_consensus.cc:3037] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:13.156473 9419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32407ee797ba4359a5f79513e69741ad" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
I20240426 09:54:13.159902 9419 raft_consensus.cc:3037] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:13.162338 9495 raft_consensus.cc:2454] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in term 1.
I20240426 09:54:13.163048 9619 leader_election.cc:288] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 election: Requested vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215)
I20240426 09:54:13.163408 9532 leader_election.cc:302] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653; no voters:
I20240426 09:54:13.164188 9619 raft_consensus.cc:2781] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:13.164592 9419 raft_consensus.cc:2454] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in term 1.
I20240426 09:54:13.164625 9619 raft_consensus.cc:686] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 LEADER]: Becoming Leader. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Running, Role: LEADER
I20240426 09:54:13.165637 9619 consensus_queue.cc:237] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.172178 9318 catalog_manager.cc:5484] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131). New cstate: current_term: 1 leader_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:13.245442 9643 consensus_queue.cc:1035] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.254945 9615 consensus_queue.cc:1035] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.260457 9419 raft_consensus.cc:3037] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:13.266563 9615 consensus_queue.cc:1035] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.275243 9644 consensus_queue.cc:1035] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.278473 9661 consensus_queue.cc:1035] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.289188 9644 raft_consensus.cc:482] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:13.289654 9644 raft_consensus.cc:504] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.289741 9614 consensus_queue.cc:1035] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.291369 9644 leader_election.cc:288] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:13.292768 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" is_pre_election: true
I20240426 09:54:13.293452 9567 raft_consensus.cc:2379] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88b95908620241b7b6992d56d159a653 in current term 1: Already voted for candidate a51b16892b1f45a6933030491fcc05aa in this term.
I20240426 09:54:13.293733 9419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51b16892b1f45a6933030491fcc05aa" is_pre_election: true
I20240426 09:54:13.295195 9461 leader_election.cc:302] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b95908620241b7b6992d56d159a653; no voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, a51b16892b1f45a6933030491fcc05aa
I20240426 09:54:13.295748 9644 raft_consensus.cc:3037] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:13.301229 9644 raft_consensus.cc:2726] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:13.315240 9614 consensus_queue.cc:1035] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.319551 9495 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "87f66b3dfc3d4b5abb5726efb3d225f1"
dest_uuid: "88b95908620241b7b6992d56d159a653"
mode: GRACEFUL
new_leader_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:48380
I20240426 09:54:13.320358 9495 raft_consensus.cc:595] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Received request to transfer leadership to TS a51b16892b1f45a6933030491fcc05aa
I20240426 09:54:13.328961 9614 consensus_queue.cc:1035] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.337344 9567 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99873d472db84e08889ea87be1b4d6df"
dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
mode: GRACEFUL
new_leader_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:36772
I20240426 09:54:13.337965 9567 raft_consensus.cc:595] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 LEADER]: Received request to transfer leadership to TS a51b16892b1f45a6933030491fcc05aa
I20240426 09:54:13.341146 9644 consensus_queue.cc:1035] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.343875 9495 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3db2ebff0e144fc1a1a5144e9c6f7e60"
dest_uuid: "88b95908620241b7b6992d56d159a653"
mode: GRACEFUL
new_leader_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:48380
I20240426 09:54:13.344295 9495 raft_consensus.cc:595] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Received request to transfer leadership to TS a51b16892b1f45a6933030491fcc05aa
I20240426 09:54:13.350576 9495 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b488843f1cb64132982e15fc409939fb"
dest_uuid: "88b95908620241b7b6992d56d159a653"
mode: GRACEFUL
new_leader_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:48380
I20240426 09:54:13.351130 9495 raft_consensus.cc:595] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Received request to transfer leadership to TS a51b16892b1f45a6933030491fcc05aa
I20240426 09:54:13.352339 9643 consensus_queue.cc:1035] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.357761 9495 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c"
dest_uuid: "88b95908620241b7b6992d56d159a653"
mode: GRACEFUL
new_leader_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:48380
I20240426 09:54:13.358326 9495 raft_consensus.cc:595] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Received request to transfer leadership to TS a51b16892b1f45a6933030491fcc05aa
I20240426 09:54:13.370980 9419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1990a88d033491bbf9904c47e91a23d"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
mode: GRACEFUL
new_leader_uuid: "88b95908620241b7b6992d56d159a653"
from {username='slave'} at 127.0.0.1:55558
I20240426 09:54:13.371448 9419 raft_consensus.cc:595] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 1 LEADER]: Received request to transfer leadership to TS 88b95908620241b7b6992d56d159a653
I20240426 09:54:13.375277 9419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
mode: GRACEFUL
new_leader_uuid: "88b95908620241b7b6992d56d159a653"
from {username='slave'} at 127.0.0.1:55558
I20240426 09:54:13.375679 9419 raft_consensus.cc:595] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 1 LEADER]: Received request to transfer leadership to TS 88b95908620241b7b6992d56d159a653
I20240426 09:54:13.381299 9567 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32407ee797ba4359a5f79513e69741ad"
dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
mode: GRACEFUL
new_leader_uuid: "88b95908620241b7b6992d56d159a653"
from {username='slave'} at 127.0.0.1:36772
I20240426 09:54:13.381747 9567 raft_consensus.cc:595] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 LEADER]: Received request to transfer leadership to TS 88b95908620241b7b6992d56d159a653
I20240426 09:54:13.381938 9614 consensus_queue.cc:1035] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.385696 9419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff15116e7e7546b1b7b04c5b227e503e"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
mode: GRACEFUL
new_leader_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
from {username='slave'} at 127.0.0.1:55558
I20240426 09:54:13.386234 9419 raft_consensus.cc:595] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 1 LEADER]: Received request to transfer leadership to TS 7b4e36cec98d4a4eb5e1f6c5ab762f19
I20240426 09:54:13.388118 9651 raft_consensus.cc:979] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa: : Instructing follower 88b95908620241b7b6992d56d159a653 to start an election
I20240426 09:54:13.388545 9661 raft_consensus.cc:1067] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 1 LEADER]: Signalling peer 88b95908620241b7b6992d56d159a653 to start an election
I20240426 09:54:13.390440 9494 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1990a88d033491bbf9904c47e91a23d"
dest_uuid: "88b95908620241b7b6992d56d159a653"
from {username='slave'} at 127.0.0.1:48370
I20240426 09:54:13.390961 9494 raft_consensus.cc:482] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:13.391311 9494 raft_consensus.cc:3037] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.393090 9661 consensus_queue.cc:1035] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.401305 9494 raft_consensus.cc:504] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.403162 9494 leader_election.cc:288] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 2 election: Requested vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:13.404253 9419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1990a88d033491bbf9904c47e91a23d" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
I20240426 09:54:13.404804 9419 raft_consensus.cc:3032] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:54:13.405086 9419 raft_consensus.cc:729] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:13.405702 9419 consensus_queue.cc:260] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.406551 9419 raft_consensus.cc:3037] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.406885 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1990a88d033491bbf9904c47e91a23d" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:13.407367 9567 raft_consensus.cc:3037] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.410655 9419 raft_consensus.cc:2454] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 2.
I20240426 09:54:13.411186 9567 raft_consensus.cc:2454] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 2.
I20240426 09:54:13.411520 9461 leader_election.cc:302] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:13.412154 9643 raft_consensus.cc:2781] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:54:13.412494 9643 raft_consensus.cc:686] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 2 LEADER]: Becoming Leader. State: Replica: 88b95908620241b7b6992d56d159a653, State: Running, Role: LEADER
I20240426 09:54:13.413161 9643 consensus_queue.cc:237] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.418483 9316 catalog_manager.cc:5484] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 reported cstate change: term changed from 1 to 2, leader changed from a51b16892b1f45a6933030491fcc05aa (127.8.141.129) to 88b95908620241b7b6992d56d159a653 (127.8.141.130). New cstate: current_term: 2 leader_uuid: "88b95908620241b7b6992d56d159a653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:13.430208 9643 consensus_queue.cc:1035] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.435509 9615 raft_consensus.cc:979] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653: : Instructing follower a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:13.435916 9639 raft_consensus.cc:1067] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Signalling peer a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:13.440137 9419 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "87f66b3dfc3d4b5abb5726efb3d225f1"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:55546
I20240426 09:54:13.440744 9419 raft_consensus.cc:482] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:13.441179 9419 raft_consensus.cc:3037] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.442414 9639 consensus_queue.cc:1035] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.446653 9419 raft_consensus.cc:504] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } }
I20240426 09:54:13.448571 9419 leader_election.cc:288] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 2 election: Requested vote from peers 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209), 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215)
I20240426 09:54:13.449187 9568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f66b3dfc3d4b5abb5726efb3d225f1" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:13.449729 9568 raft_consensus.cc:3037] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.454195 9568 raft_consensus.cc:2454] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 2.
I20240426 09:54:13.455260 9386 leader_election.cc:302] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:13.455899 9661 raft_consensus.cc:2781] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:54:13.456140 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f66b3dfc3d4b5abb5726efb3d225f1" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:13.456379 9661 raft_consensus.cc:686] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Becoming Leader. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:13.456698 9494 raft_consensus.cc:3032] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:54:13.457013 9494 raft_consensus.cc:729] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b95908620241b7b6992d56d159a653, State: Running, Role: LEADER
I20240426 09:54:13.457173 9661 consensus_queue.cc:237] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } }
I20240426 09:54:13.457760 9494 consensus_queue.cc:260] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } }
I20240426 09:54:13.458994 9494 raft_consensus.cc:3037] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.463179 9494 raft_consensus.cc:2454] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 2.
I20240426 09:54:13.463076 9318 catalog_manager.cc:5484] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa reported cstate change: term changed from 1 to 2, leader changed from 88b95908620241b7b6992d56d159a653 (127.8.141.130) to a51b16892b1f45a6933030491fcc05aa (127.8.141.129). New cstate: current_term: 2 leader_uuid: "a51b16892b1f45a6933030491fcc05aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:13.606213 9615 raft_consensus.cc:979] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653: : Instructing follower a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:13.606609 9639 raft_consensus.cc:1067] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Signalling peer a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:13.607833 9419 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3db2ebff0e144fc1a1a5144e9c6f7e60"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:55546
I20240426 09:54:13.608253 9419 raft_consensus.cc:482] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:13.608507 9419 raft_consensus.cc:3037] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.612663 9419 raft_consensus.cc:504] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.613963 9419 leader_election.cc:288] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 2 election: Requested vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:13.614564 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db2ebff0e144fc1a1a5144e9c6f7e60" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:13.615000 9568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db2ebff0e144fc1a1a5144e9c6f7e60" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:13.615103 9494 raft_consensus.cc:3032] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:54:13.615435 9494 raft_consensus.cc:729] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b95908620241b7b6992d56d159a653, State: Running, Role: LEADER
I20240426 09:54:13.615450 9568 raft_consensus.cc:3037] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.616092 9494 consensus_queue.cc:260] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.617049 9494 raft_consensus.cc:3037] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.619822 9568 raft_consensus.cc:2454] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 2.
I20240426 09:54:13.620683 9386 leader_election.cc:302] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:13.621285 9661 raft_consensus.cc:2781] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:54:13.621361 9494 raft_consensus.cc:2454] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 2.
I20240426 09:54:13.621838 9661 raft_consensus.cc:686] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Becoming Leader. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:13.622623 9661 consensus_queue.cc:237] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.627930 9318 catalog_manager.cc:5484] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa reported cstate change: term changed from 1 to 2, leader changed from 88b95908620241b7b6992d56d159a653 (127.8.141.130) to a51b16892b1f45a6933030491fcc05aa (127.8.141.129). New cstate: current_term: 2 leader_uuid: "a51b16892b1f45a6933030491fcc05aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:13.657389 9619 consensus_queue.cc:1035] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.667419 9646 consensus_queue.cc:1035] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:13.674088 9619 raft_consensus.cc:979] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19: : Instructing follower 88b95908620241b7b6992d56d159a653 to start an election
I20240426 09:54:13.674525 9646 raft_consensus.cc:1067] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 LEADER]: Signalling peer 88b95908620241b7b6992d56d159a653 to start an election
I20240426 09:54:13.675827 9494 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32407ee797ba4359a5f79513e69741ad"
dest_uuid: "88b95908620241b7b6992d56d159a653"
from {username='slave'} at 127.0.0.1:48376
I20240426 09:54:13.676249 9494 raft_consensus.cc:482] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:13.676509 9494 raft_consensus.cc:3037] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.680663 9494 raft_consensus.cc:504] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.681985 9494 leader_election.cc:288] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 2 election: Requested vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:13.682861 9419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32407ee797ba4359a5f79513e69741ad" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
I20240426 09:54:13.683069 9568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32407ee797ba4359a5f79513e69741ad" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:13.683432 9419 raft_consensus.cc:3037] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.683585 9568 raft_consensus.cc:3032] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:54:13.683954 9568 raft_consensus.cc:729] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Running, Role: LEADER
I20240426 09:54:13.684689 9568 consensus_queue.cc:260] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.685900 9568 raft_consensus.cc:3037] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.688058 9419 raft_consensus.cc:2454] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 2.
I20240426 09:54:13.688910 9461 leader_election.cc:302] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:13.689499 9615 raft_consensus.cc:2781] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:54:13.689962 9615 raft_consensus.cc:686] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [term 2 LEADER]: Becoming Leader. State: Replica: 88b95908620241b7b6992d56d159a653, State: Running, Role: LEADER
I20240426 09:54:13.691001 9568 raft_consensus.cc:2454] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 2.
I20240426 09:54:13.690717 9615 consensus_queue.cc:237] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.696869 9316 catalog_manager.cc:5484] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 reported cstate change: term changed from 1 to 2, leader changed from 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131) to 88b95908620241b7b6992d56d159a653 (127.8.141.130). New cstate: current_term: 2 leader_uuid: "88b95908620241b7b6992d56d159a653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:13.755705 9619 raft_consensus.cc:979] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19: : Instructing follower a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:13.756177 9701 raft_consensus.cc:1067] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 LEADER]: Signalling peer a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:13.757761 9419 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99873d472db84e08889ea87be1b4d6df"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:55554
I20240426 09:54:13.758203 9419 raft_consensus.cc:482] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:13.758473 9419 raft_consensus.cc:3037] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.762604 9419 raft_consensus.cc:504] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.763954 9419 leader_election.cc:288] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 2 election: Requested vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:13.764685 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99873d472db84e08889ea87be1b4d6df" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:13.764865 9568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99873d472db84e08889ea87be1b4d6df" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:13.765277 9494 raft_consensus.cc:3037] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.765368 9568 raft_consensus.cc:3032] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:54:13.765729 9568 raft_consensus.cc:729] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Running, Role: LEADER
I20240426 09:54:13.766454 9568 consensus_queue.cc:260] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.767676 9568 raft_consensus.cc:3037] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.771405 9494 raft_consensus.cc:2454] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 2.
I20240426 09:54:13.772358 9383 leader_election.cc:302] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:13.772969 9661 raft_consensus.cc:2781] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:54:13.773311 9568 raft_consensus.cc:2454] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 2.
I20240426 09:54:13.773418 9661 raft_consensus.cc:686] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Becoming Leader. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:13.774219 9661 consensus_queue.cc:237] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.779770 9316 catalog_manager.cc:5484] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa reported cstate change: term changed from 1 to 2, leader changed from 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131) to a51b16892b1f45a6933030491fcc05aa (127.8.141.129). New cstate: current_term: 2 leader_uuid: "a51b16892b1f45a6933030491fcc05aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:13.813833 9639 raft_consensus.cc:979] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653: : Instructing follower a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:13.814152 9615 raft_consensus.cc:1067] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Signalling peer a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:13.815266 9419 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b488843f1cb64132982e15fc409939fb"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:55546
I20240426 09:54:13.815639 9419 raft_consensus.cc:482] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:13.815869 9419 raft_consensus.cc:3037] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.820024 9419 raft_consensus.cc:504] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.821326 9419 leader_election.cc:288] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 2 election: Requested vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:13.821956 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b488843f1cb64132982e15fc409939fb" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:13.822232 9568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b488843f1cb64132982e15fc409939fb" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:13.822461 9494 raft_consensus.cc:3032] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:54:13.822680 9568 raft_consensus.cc:3037] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.822741 9494 raft_consensus.cc:729] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b95908620241b7b6992d56d159a653, State: Running, Role: LEADER
I20240426 09:54:13.823416 9494 consensus_queue.cc:260] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.824244 9494 raft_consensus.cc:3037] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.826745 9568 raft_consensus.cc:2454] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 2.
I20240426 09:54:13.827543 9386 leader_election.cc:302] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:13.828260 9661 raft_consensus.cc:2781] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:54:13.828418 9494 raft_consensus.cc:2454] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 2.
I20240426 09:54:13.828640 9661 raft_consensus.cc:686] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Becoming Leader. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:13.829440 9661 consensus_queue.cc:237] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.835239 9316 catalog_manager.cc:5484] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa reported cstate change: term changed from 1 to 2, leader changed from 88b95908620241b7b6992d56d159a653 (127.8.141.130) to a51b16892b1f45a6933030491fcc05aa (127.8.141.129). New cstate: current_term: 2 leader_uuid: "a51b16892b1f45a6933030491fcc05aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:13.836776 9494 raft_consensus.cc:1261] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:13.839270 9661 consensus_queue.cc:1035] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:13.842355 9568 raft_consensus.cc:1261] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Refusing update from remote peer 88b95908620241b7b6992d56d159a653: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:13.843464 9615 consensus_queue.cc:1035] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:13.848994 9567 raft_consensus.cc:1261] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:13.850387 9702 consensus_queue.cc:1035] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:13.859265 9419 raft_consensus.cc:1261] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Refusing update from remote peer 88b95908620241b7b6992d56d159a653: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:13.861073 9639 consensus_queue.cc:1035] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:13.872900 9643 raft_consensus.cc:979] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653: : Instructing follower a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:13.873292 9639 raft_consensus.cc:1067] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Signalling peer a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:13.874634 9614 raft_consensus.cc:979] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa: : Instructing follower 7b4e36cec98d4a4eb5e1f6c5ab762f19 to start an election
I20240426 09:54:13.874989 9419 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:55546
I20240426 09:54:13.875072 9702 raft_consensus.cc:1067] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 1 LEADER]: Signalling peer 7b4e36cec98d4a4eb5e1f6c5ab762f19 to start an election
I20240426 09:54:13.875444 9419 raft_consensus.cc:482] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:13.875725 9419 raft_consensus.cc:3037] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.876356 9568 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff15116e7e7546b1b7b04c5b227e503e"
dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
from {username='slave'} at 127.0.0.1:36752
I20240426 09:54:13.876751 9568 raft_consensus.cc:482] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:13.877007 9568 raft_consensus.cc:3037] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.880092 9419 raft_consensus.cc:504] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.881012 9568 raft_consensus.cc:504] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.881443 9419 leader_election.cc:288] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 2 election: Requested vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:13.882279 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:13.882386 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:13.882790 9568 leader_election.cc:288] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 2 election: Requested vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215)
I20240426 09:54:13.882927 9494 raft_consensus.cc:3032] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:54:13.883296 9494 raft_consensus.cc:729] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b95908620241b7b6992d56d159a653, State: Running, Role: LEADER
I20240426 09:54:13.882993 9567 raft_consensus.cc:3037] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.883703 9495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff15116e7e7546b1b7b04c5b227e503e" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:13.883960 9494 consensus_queue.cc:260] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.884267 9495 raft_consensus.cc:3037] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.884877 9494 raft_consensus.cc:3037] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.887605 9419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff15116e7e7546b1b7b04c5b227e503e" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
I20240426 09:54:13.888310 9419 raft_consensus.cc:3032] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:54:13.888602 9567 raft_consensus.cc:2454] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 2.
I20240426 09:54:13.888773 9419 raft_consensus.cc:729] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:13.889614 9386 leader_election.cc:302] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:13.889604 9419 consensus_queue.cc:260] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.890235 9495 raft_consensus.cc:2454] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in term 2.
I20240426 09:54:13.890535 9702 raft_consensus.cc:2781] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:54:13.891042 9702 raft_consensus.cc:686] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Becoming Leader. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:13.891112 9419 raft_consensus.cc:3037] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.891307 9532 leader_election.cc:302] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653; no voters:
I20240426 09:54:13.891608 9494 raft_consensus.cc:2454] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 2.
I20240426 09:54:13.892215 9701 raft_consensus.cc:2781] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:54:13.892659 9701 raft_consensus.cc:686] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 LEADER]: Becoming Leader. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Running, Role: LEADER
I20240426 09:54:13.893399 9701 consensus_queue.cc:237] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.891850 9702 consensus_queue.cc:237] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.897642 9419 raft_consensus.cc:2454] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in term 2.
I20240426 09:54:13.900494 9315 catalog_manager.cc:5484] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 reported cstate change: term changed from 1 to 2, leader changed from a51b16892b1f45a6933030491fcc05aa (127.8.141.129) to 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131). New cstate: current_term: 2 leader_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:13.900642 9316 catalog_manager.cc:5484] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa reported cstate change: term changed from 1 to 2, leader changed from 88b95908620241b7b6992d56d159a653 (127.8.141.130) to a51b16892b1f45a6933030491fcc05aa (127.8.141.129). New cstate: current_term: 2 leader_uuid: "a51b16892b1f45a6933030491fcc05aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:13.951146 9614 raft_consensus.cc:979] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa: : Instructing follower 88b95908620241b7b6992d56d159a653 to start an election
I20240426 09:54:13.951659 9702 raft_consensus.cc:1067] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 1 LEADER]: Signalling peer 88b95908620241b7b6992d56d159a653 to start an election
I20240426 09:54:13.952915 9494 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e"
dest_uuid: "88b95908620241b7b6992d56d159a653"
from {username='slave'} at 127.0.0.1:48370
I20240426 09:54:13.953457 9494 raft_consensus.cc:482] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:13.953819 9494 raft_consensus.cc:3037] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.958148 9494 raft_consensus.cc:504] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.959460 9494 leader_election.cc:288] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 2 election: Requested vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:13.960129 9419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
I20240426 09:54:13.960328 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:13.960770 9419 raft_consensus.cc:3032] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:54:13.960877 9567 raft_consensus.cc:3037] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.961143 9419 raft_consensus.cc:729] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:13.961938 9419 consensus_queue.cc:260] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.963212 9419 raft_consensus.cc:3037] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:13.967121 9567 raft_consensus.cc:2454] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 2.
I20240426 09:54:13.968151 9461 leader_election.cc:302] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653; no voters:
I20240426 09:54:13.968894 9639 raft_consensus.cc:2781] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:54:13.969188 9419 raft_consensus.cc:2454] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 2.
I20240426 09:54:13.969338 9639 raft_consensus.cc:686] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 2 LEADER]: Becoming Leader. State: Replica: 88b95908620241b7b6992d56d159a653, State: Running, Role: LEADER
I20240426 09:54:13.970155 9639 consensus_queue.cc:237] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:13.977317 9316 catalog_manager.cc:5484] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 reported cstate change: term changed from 1 to 2, leader changed from a51b16892b1f45a6933030491fcc05aa (127.8.141.129) to 88b95908620241b7b6992d56d159a653 (127.8.141.130). New cstate: current_term: 2 leader_uuid: "88b95908620241b7b6992d56d159a653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:14.067801 9567 raft_consensus.cc:1261] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:14.068881 9702 consensus_queue.cc:1035] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:14.075259 9494 raft_consensus.cc:1261] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:14.076455 9651 consensus_queue.cc:1035] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:14.174096 9494 raft_consensus.cc:1261] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:14.175251 9651 consensus_queue.cc:1035] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:14.182487 9567 raft_consensus.cc:1261] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:14.183851 9702 consensus_queue.cc:1035] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:14.187615 9568 raft_consensus.cc:1261] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Refusing update from remote peer 88b95908620241b7b6992d56d159a653: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:14.189338 9639 consensus_queue.cc:1035] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:14.195859 9419 raft_consensus.cc:1261] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Refusing update from remote peer 88b95908620241b7b6992d56d159a653: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:14.198007 9639 consensus_queue.cc:1035] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:14.271742 9419 raft_consensus.cc:1261] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Refusing update from remote peer 7b4e36cec98d4a4eb5e1f6c5ab762f19: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:14.272729 9701 consensus_queue.cc:1035] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:14.276700 9494 raft_consensus.cc:1261] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:14.278991 9494 raft_consensus.cc:1261] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Refusing update from remote peer 7b4e36cec98d4a4eb5e1f6c5ab762f19: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:14.279330 9702 consensus_queue.cc:1035] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:14.280504 9619 consensus_queue.cc:1035] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:14.290434 9568 raft_consensus.cc:1261] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:14.291658 9661 consensus_queue.cc:1035] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:14.306175 9568 raft_consensus.cc:1261] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:14.307153 9702 consensus_queue.cc:1035] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:14.313971 9494 raft_consensus.cc:1261] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:14.315192 9614 consensus_queue.cc:1035] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:14.442368 9419 raft_consensus.cc:1261] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Refusing update from remote peer 88b95908620241b7b6992d56d159a653: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:14.443362 9639 consensus_queue.cc:1035] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:14.450275 9568 raft_consensus.cc:1261] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Refusing update from remote peer 88b95908620241b7b6992d56d159a653: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:14.451869 9615 consensus_queue.cc:1035] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 09:54:16.407277 8758 auto_leader_rebalancer-test.cc:331] The leader distribution is
I20240426 09:54:16.407877 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:16.421736 9418 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3db2ebff0e144fc1a1a5144e9c6f7e60"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
mode: GRACEFUL
new_leader_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
from {username='slave'} at 127.0.0.1:43014
I20240426 09:54:16.422204 9418 raft_consensus.cc:595] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Received request to transfer leadership to TS 7b4e36cec98d4a4eb5e1f6c5ab762f19
I20240426 09:54:16.424098 9418 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
mode: GRACEFUL
new_leader_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
from {username='slave'} at 127.0.0.1:43014
I20240426 09:54:16.424515 9418 raft_consensus.cc:595] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Received request to transfer leadership to TS 7b4e36cec98d4a4eb5e1f6c5ab762f19
I20240426 09:54:16.425316 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240426 09:54:16.425657 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:16.560506 9614 raft_consensus.cc:979] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa: : Instructing follower 7b4e36cec98d4a4eb5e1f6c5ab762f19 to start an election
I20240426 09:54:16.560894 9614 raft_consensus.cc:1067] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Signalling peer 7b4e36cec98d4a4eb5e1f6c5ab762f19 to start an election
I20240426 09:54:16.561955 9567 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c"
dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
from {username='slave'} at 127.0.0.1:36752
I20240426 09:54:16.562371 9567 raft_consensus.cc:482] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:16.562636 9567 raft_consensus.cc:3037] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:16.566731 9567 raft_consensus.cc:504] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:16.568176 9567 leader_election.cc:288] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 3 election: Requested vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215)
I20240426 09:54:16.568818 9418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
I20240426 09:54:16.569096 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:16.569330 9418 raft_consensus.cc:3032] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:54:16.569605 9494 raft_consensus.cc:3037] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:16.569602 9418 raft_consensus.cc:729] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:16.570339 9418 consensus_queue.cc:260] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:16.571228 9418 raft_consensus.cc:3037] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:16.573860 9494 raft_consensus.cc:2454] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in term 3.
I20240426 09:54:16.574824 9532 leader_election.cc:302] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653; no voters:
I20240426 09:54:16.575471 9619 raft_consensus.cc:2781] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:54:16.575618 9418 raft_consensus.cc:2454] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in term 3.
I20240426 09:54:16.576812 9619 raft_consensus.cc:686] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 LEADER]: Becoming Leader. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Running, Role: LEADER
I20240426 09:54:16.577201 9715 raft_consensus.cc:979] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa: : Instructing follower 7b4e36cec98d4a4eb5e1f6c5ab762f19 to start an election
I20240426 09:54:16.577616 9614 raft_consensus.cc:1067] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Signalling peer 7b4e36cec98d4a4eb5e1f6c5ab762f19 to start an election
I20240426 09:54:16.577639 9619 consensus_queue.cc:237] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:16.579145 9567 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3db2ebff0e144fc1a1a5144e9c6f7e60"
dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
from {username='slave'} at 127.0.0.1:36752
I20240426 09:54:16.579553 9567 raft_consensus.cc:482] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:16.579805 9567 raft_consensus.cc:3037] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:16.584491 9567 raft_consensus.cc:504] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:16.584563 9316 catalog_manager.cc:5484] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 reported cstate change: term changed from 2 to 3, leader changed from a51b16892b1f45a6933030491fcc05aa (127.8.141.129) to 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131). New cstate: current_term: 3 leader_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:16.585911 9567 leader_election.cc:288] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 3 election: Requested vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215)
I20240426 09:54:16.586861 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db2ebff0e144fc1a1a5144e9c6f7e60" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:16.586889 9418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db2ebff0e144fc1a1a5144e9c6f7e60" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
I20240426 09:54:16.587646 9418 raft_consensus.cc:3032] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:54:16.587615 9494 raft_consensus.cc:3037] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:16.588042 9418 raft_consensus.cc:729] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:16.588738 9418 consensus_queue.cc:260] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:16.589573 9418 raft_consensus.cc:3037] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:16.594017 9494 raft_consensus.cc:2454] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in term 3.
I20240426 09:54:16.594485 9418 raft_consensus.cc:2454] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in term 3.
I20240426 09:54:16.595059 9532 leader_election.cc:302] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653; no voters:
I20240426 09:54:16.595811 9619 raft_consensus.cc:2781] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:54:16.596164 9619 raft_consensus.cc:686] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 LEADER]: Becoming Leader. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Running, Role: LEADER
I20240426 09:54:16.596782 9619 consensus_queue.cc:237] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:16.606792 9316 catalog_manager.cc:5484] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 reported cstate change: term changed from 2 to 3, leader changed from a51b16892b1f45a6933030491fcc05aa (127.8.141.129) to 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131). New cstate: current_term: 3 leader_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:17.040550 9418 raft_consensus.cc:1261] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Refusing update from remote peer 7b4e36cec98d4a4eb5e1f6c5ab762f19: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:54:17.041618 9619 consensus_queue.cc:1035] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:17.049249 9494 raft_consensus.cc:1261] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Refusing update from remote peer 7b4e36cec98d4a4eb5e1f6c5ab762f19: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:54:17.050717 9716 consensus_queue.cc:1035] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:17.157366 9494 raft_consensus.cc:1261] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Refusing update from remote peer 7b4e36cec98d4a4eb5e1f6c5ab762f19: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:54:17.158296 9716 consensus_queue.cc:1035] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:17.166364 9418 raft_consensus.cc:1261] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Refusing update from remote peer 7b4e36cec98d4a4eb5e1f6c5ab762f19: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:54:17.167383 9619 consensus_queue.cc:1035] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:17.426283 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:17.431655 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:17.432152 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:18.433082 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:18.438472 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:18.438863 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:19.439530 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:19.445101 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:19.445506 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:20.446138 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:20.452158 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:20.452517 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:21.453197 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:21.458092 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:21.458499 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:22.508405 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:22.513293 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:22.513655 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240426 09:54:23.033969 9441 debug-util.cc:398] Leaking SignalData structure 0x7b08001e6000 after lost signal to thread 8761
W20240426 09:54:23.034649 9441 debug-util.cc:398] Leaking SignalData structure 0x7b08001e61c0 after lost signal to thread 9352
W20240426 09:54:23.035445 9441 debug-util.cc:398] Leaking SignalData structure 0x7b08000a1680 after lost signal to thread 9444
W20240426 09:54:23.036226 9441 debug-util.cc:398] Leaking SignalData structure 0x7b08000dcb20 after lost signal to thread 9519
W20240426 09:54:23.036975 9441 debug-util.cc:398] Leaking SignalData structure 0x7b08001b4c40 after lost signal to thread 9593
I20240426 09:54:23.514370 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:23.520550 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:23.520917 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:24.521615 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:24.526937 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:24.527297 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:25.527958 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:25.532976 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:25.533429 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:26.555389 8758 auto_leader_rebalancer-test.cc:350] The leader distribution is
I20240426 09:54:26.562923 9567 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3db2ebff0e144fc1a1a5144e9c6f7e60"
dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
mode: GRACEFUL
new_leader_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:36772
I20240426 09:54:26.563519 9567 raft_consensus.cc:595] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 LEADER]: Received request to transfer leadership to TS a51b16892b1f45a6933030491fcc05aa
I20240426 09:54:26.569275 9567 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c"
dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
mode: GRACEFUL
new_leader_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:36772
I20240426 09:54:26.569648 9567 raft_consensus.cc:595] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 LEADER]: Received request to transfer leadership to TS a51b16892b1f45a6933030491fcc05aa
I20240426 09:54:26.573282 9493 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1990a88d033491bbf9904c47e91a23d"
dest_uuid: "88b95908620241b7b6992d56d159a653"
mode: GRACEFUL
new_leader_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:48380
I20240426 09:54:26.573688 9493 raft_consensus.cc:595] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 2 LEADER]: Received request to transfer leadership to TS a51b16892b1f45a6933030491fcc05aa
I20240426 09:54:26.577284 9493 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e"
dest_uuid: "88b95908620241b7b6992d56d159a653"
mode: GRACEFUL
new_leader_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:48380
I20240426 09:54:26.577700 9493 raft_consensus.cc:595] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 2 LEADER]: Received request to transfer leadership to TS a51b16892b1f45a6933030491fcc05aa
I20240426 09:54:26.581135 9493 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "313a150ad860483a978ae5584f5554e0"
dest_uuid: "88b95908620241b7b6992d56d159a653"
mode: GRACEFUL
new_leader_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:48380
I20240426 09:54:26.581516 9493 raft_consensus.cc:595] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Received request to transfer leadership to TS a51b16892b1f45a6933030491fcc05aa
I20240426 09:54:26.674964 9741 raft_consensus.cc:979] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19: : Instructing follower a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:26.675499 9736 raft_consensus.cc:1067] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 LEADER]: Signalling peer a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:26.676820 9419 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:55554
I20240426 09:54:26.677227 9419 raft_consensus.cc:482] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:26.677543 9419 raft_consensus.cc:3037] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:54:26.681975 9419 raft_consensus.cc:504] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:26.683400 9419 leader_election.cc:288] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 4 election: Requested vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:26.684244 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:26.684154 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:26.684815 9567 raft_consensus.cc:3032] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:54:26.684881 9493 raft_consensus.cc:3037] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:54:26.685078 9567 raft_consensus.cc:729] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Running, Role: LEADER
I20240426 09:54:26.685721 9567 consensus_queue.cc:260] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:26.686548 9567 raft_consensus.cc:3037] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:54:26.689601 9493 raft_consensus.cc:2454] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 4.
I20240426 09:54:26.690577 9383 leader_election.cc:302] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:26.690739 9567 raft_consensus.cc:2454] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 4.
I20240426 09:54:26.691433 9733 raft_consensus.cc:2781] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:54:26.691859 9733 raft_consensus.cc:686] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 4 LEADER]: Becoming Leader. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:26.692591 9733 consensus_queue.cc:237] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:26.700685 9318 catalog_manager.cc:5484] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa reported cstate change: term changed from 3 to 4, leader changed from 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131) to a51b16892b1f45a6933030491fcc05aa (127.8.141.129). New cstate: current_term: 4 leader_uuid: "a51b16892b1f45a6933030491fcc05aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:26.750173 9737 raft_consensus.cc:979] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653: : Instructing follower a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:26.750638 9747 raft_consensus.cc:1067] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 2 LEADER]: Signalling peer a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:26.751827 9419 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1990a88d033491bbf9904c47e91a23d"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:55546
I20240426 09:54:26.752240 9419 raft_consensus.cc:482] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:26.752539 9419 raft_consensus.cc:3037] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:26.756528 9419 raft_consensus.cc:504] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:26.758034 9419 leader_election.cc:288] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 3 election: Requested vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:26.759148 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1990a88d033491bbf9904c47e91a23d" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:26.759805 9493 raft_consensus.cc:3032] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:54:26.760128 9493 raft_consensus.cc:729] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 88b95908620241b7b6992d56d159a653, State: Running, Role: LEADER
I20240426 09:54:26.760802 9493 consensus_queue.cc:260] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:26.761983 9493 raft_consensus.cc:3037] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:26.767832 9493 raft_consensus.cc:2454] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 3.
I20240426 09:54:26.768959 9383 leader_election.cc:302] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:26.759219 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1990a88d033491bbf9904c47e91a23d" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:26.769706 9567 raft_consensus.cc:3037] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:26.769790 9733 raft_consensus.cc:2781] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:54:26.770226 9733 raft_consensus.cc:686] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 3 LEADER]: Becoming Leader. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:26.771008 9733 consensus_queue.cc:237] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:26.773633 9567 raft_consensus.cc:2454] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 3.
I20240426 09:54:26.777663 9318 catalog_manager.cc:5484] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa reported cstate change: term changed from 2 to 3, leader changed from 88b95908620241b7b6992d56d159a653 (127.8.141.130) to a51b16892b1f45a6933030491fcc05aa (127.8.141.129). New cstate: current_term: 3 leader_uuid: "a51b16892b1f45a6933030491fcc05aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:26.784404 9737 raft_consensus.cc:979] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653: : Instructing follower a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:26.784803 9737 raft_consensus.cc:1067] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Signalling peer a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:26.785948 9419 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "313a150ad860483a978ae5584f5554e0"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:55546
I20240426 09:54:26.786294 9419 raft_consensus.cc:482] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:26.786525 9419 raft_consensus.cc:3037] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:26.790081 9419 raft_consensus.cc:504] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:26.791290 9419 leader_election.cc:288] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 2 election: Requested vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:26.791962 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313a150ad860483a978ae5584f5554e0" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:26.792042 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313a150ad860483a978ae5584f5554e0" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:26.792635 9567 raft_consensus.cc:3037] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:26.792598 9493 raft_consensus.cc:3032] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:54:26.793097 9493 raft_consensus.cc:729] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b95908620241b7b6992d56d159a653, State: Running, Role: LEADER
I20240426 09:54:26.793835 9493 consensus_queue.cc:260] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:26.795107 9493 raft_consensus.cc:3037] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:26.798138 9567 raft_consensus.cc:2454] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 2.
I20240426 09:54:26.799047 9386 leader_election.cc:302] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:26.799603 9733 raft_consensus.cc:2781] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:54:26.799935 9733 raft_consensus.cc:686] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Becoming Leader. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:26.800380 9493 raft_consensus.cc:2454] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 2.
I20240426 09:54:26.800580 9733 consensus_queue.cc:237] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:26.806306 9318 catalog_manager.cc:5484] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa reported cstate change: term changed from 1 to 2, leader changed from 88b95908620241b7b6992d56d159a653 (127.8.141.130) to a51b16892b1f45a6933030491fcc05aa (127.8.141.129). New cstate: current_term: 2 leader_uuid: "a51b16892b1f45a6933030491fcc05aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:26.943048 9741 raft_consensus.cc:979] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: : Instructing follower a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:26.943451 9736 raft_consensus.cc:1067] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 LEADER]: Signalling peer a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:26.944622 9419 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3db2ebff0e144fc1a1a5144e9c6f7e60"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:55554
I20240426 09:54:26.945092 9419 raft_consensus.cc:482] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:26.945369 9419 raft_consensus.cc:3037] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:54:26.949577 9419 raft_consensus.cc:504] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:26.950856 9419 leader_election.cc:288] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 4 election: Requested vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:26.951539 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db2ebff0e144fc1a1a5144e9c6f7e60" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:26.951857 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db2ebff0e144fc1a1a5144e9c6f7e60" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:26.952172 9493 raft_consensus.cc:3037] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:54:26.952468 9567 raft_consensus.cc:3032] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:54:26.952797 9567 raft_consensus.cc:729] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Running, Role: LEADER
I20240426 09:54:26.953503 9567 consensus_queue.cc:260] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:26.954653 9567 raft_consensus.cc:3037] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:54:26.955390 9747 raft_consensus.cc:979] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653: : Instructing follower a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:26.955781 9737 raft_consensus.cc:1067] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 2 LEADER]: Signalling peer a51b16892b1f45a6933030491fcc05aa to start an election
I20240426 09:54:26.957149 9419 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
from {username='slave'} at 127.0.0.1:55546
I20240426 09:54:26.957631 9419 raft_consensus.cc:482] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:26.957872 9493 raft_consensus.cc:2454] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 4.
I20240426 09:54:26.958019 9419 raft_consensus.cc:3037] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:26.958710 9383 leader_election.cc:302] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:26.959319 9733 raft_consensus.cc:2781] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:54:26.959726 9733 raft_consensus.cc:686] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 4 LEADER]: Becoming Leader. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:26.960484 9733 consensus_queue.cc:237] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:26.960670 9567 raft_consensus.cc:2454] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 4.
I20240426 09:54:26.963070 9419 raft_consensus.cc:504] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:26.964725 9419 leader_election.cc:288] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 3 election: Requested vote from peers 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:26.965304 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:26.965823 9493 raft_consensus.cc:3032] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:54:26.965945 9567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e" candidate_uuid: "a51b16892b1f45a6933030491fcc05aa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:26.966151 9493 raft_consensus.cc:729] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 88b95908620241b7b6992d56d159a653, State: Running, Role: LEADER
I20240426 09:54:26.966390 9567 raft_consensus.cc:3037] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:26.966923 9493 consensus_queue.cc:260] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:26.968155 9493 raft_consensus.cc:3037] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:26.967904 9318 catalog_manager.cc:5484] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa reported cstate change: term changed from 3 to 4, leader changed from 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131) to a51b16892b1f45a6933030491fcc05aa (127.8.141.129). New cstate: current_term: 4 leader_uuid: "a51b16892b1f45a6933030491fcc05aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:26.969770 9567 raft_consensus.cc:2454] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 3.
I20240426 09:54:26.970515 9386 leader_election.cc:302] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:26.971295 9733 raft_consensus.cc:2781] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:54:26.971741 9733 raft_consensus.cc:686] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 3 LEADER]: Becoming Leader. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:26.972541 9733 consensus_queue.cc:237] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:26.973526 9493 raft_consensus.cc:2454] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51b16892b1f45a6933030491fcc05aa in term 3.
I20240426 09:54:26.979477 9318 catalog_manager.cc:5484] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa reported cstate change: term changed from 2 to 3, leader changed from 88b95908620241b7b6992d56d159a653 (127.8.141.130) to a51b16892b1f45a6933030491fcc05aa (127.8.141.129). New cstate: current_term: 3 leader_uuid: "a51b16892b1f45a6933030491fcc05aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:27.085552 9567 raft_consensus.cc:1261] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 4 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:54:27.086526 9733 consensus_queue.cc:1035] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:27.098685 9493 raft_consensus.cc:1261] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 4 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:54:27.100106 9740 consensus_queue.cc:1035] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:27.201503 9493 raft_consensus.cc:1261] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:27.202647 9733 consensus_queue.cc:1035] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:27.210274 9567 raft_consensus.cc:1261] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:54:27.211977 9740 consensus_queue.cc:1035] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:27.317981 9567 raft_consensus.cc:1261] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:54:27.318969 9740 consensus_queue.cc:1035] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:27.327780 9494 raft_consensus.cc:1261] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:54:27.328840 9733 consensus_queue.cc:1035] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:27.355561 9494 raft_consensus.cc:1261] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:54:27.356509 9733 consensus_queue.cc:1035] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:27.363451 9567 raft_consensus.cc:1261] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:54:27.365245 9740 consensus_queue.cc:1035] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:27.391119 9494 raft_consensus.cc:1261] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 4 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:54:27.392016 9740 consensus_queue.cc:1035] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:27.400056 9567 raft_consensus.cc:1261] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 4 FOLLOWER]: Refusing update from remote peer a51b16892b1f45a6933030491fcc05aa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:54:27.401690 9733 consensus_queue.cc:1035] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:29.606910 8758 auto_leader_rebalancer-test.cc:365] The leader distribution is
I20240426 09:54:29.607614 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:29.613919 9419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
mode: GRACEFUL
new_leader_uuid: "88b95908620241b7b6992d56d159a653"
from {username='slave'} at 127.0.0.1:43014
I20240426 09:54:29.614519 9419 raft_consensus.cc:595] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 3 LEADER]: Received request to transfer leadership to TS 88b95908620241b7b6992d56d159a653
I20240426 09:54:29.616714 9419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "87f66b3dfc3d4b5abb5726efb3d225f1"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
mode: GRACEFUL
new_leader_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
from {username='slave'} at 127.0.0.1:43014
I20240426 09:54:29.617082 9419 raft_consensus.cc:595] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Received request to transfer leadership to TS 7b4e36cec98d4a4eb5e1f6c5ab762f19
I20240426 09:54:29.618734 9419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99873d472db84e08889ea87be1b4d6df"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
mode: GRACEFUL
new_leader_uuid: "88b95908620241b7b6992d56d159a653"
from {username='slave'} at 127.0.0.1:43014
I20240426 09:54:29.619118 9419 raft_consensus.cc:595] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Received request to transfer leadership to TS 88b95908620241b7b6992d56d159a653
I20240426 09:54:29.620882 9419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c"
dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
mode: GRACEFUL
new_leader_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
from {username='slave'} at 127.0.0.1:43014
I20240426 09:54:29.621246 9419 raft_consensus.cc:595] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 4 LEADER]: Received request to transfer leadership to TS 7b4e36cec98d4a4eb5e1f6c5ab762f19
I20240426 09:54:29.622296 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 09:54:29.622853 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:29.622857 9768 raft_consensus.cc:979] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa: : Instructing follower 7b4e36cec98d4a4eb5e1f6c5ab762f19 to start an election
I20240426 09:54:29.623415 9733 raft_consensus.cc:1067] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Signalling peer 7b4e36cec98d4a4eb5e1f6c5ab762f19 to start an election
I20240426 09:54:29.624576 9567 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "87f66b3dfc3d4b5abb5726efb3d225f1"
dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
from {username='slave'} at 127.0.0.1:36752
I20240426 09:54:29.624940 9567 raft_consensus.cc:482] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:29.625222 9567 raft_consensus.cc:3037] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:29.630074 9567 raft_consensus.cc:504] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } }
I20240426 09:54:29.631906 9567 leader_election.cc:288] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 3 election: Requested vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215)
I20240426 09:54:29.632491 9419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f66b3dfc3d4b5abb5726efb3d225f1" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
I20240426 09:54:29.633059 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f66b3dfc3d4b5abb5726efb3d225f1" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:29.633198 9419 raft_consensus.cc:3032] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:54:29.633589 9494 raft_consensus.cc:3037] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:29.633579 9419 raft_consensus.cc:729] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:29.634533 9419 consensus_queue.cc:260] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } }
I20240426 09:54:29.635823 9419 raft_consensus.cc:3037] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:29.639931 9494 raft_consensus.cc:2454] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in term 3.
I20240426 09:54:29.641053 9532 leader_election.cc:302] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653; no voters:
I20240426 09:54:29.641901 9736 raft_consensus.cc:2781] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:54:29.642028 9419 raft_consensus.cc:2454] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in term 3.
I20240426 09:54:29.642869 9736 raft_consensus.cc:686] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 LEADER]: Becoming Leader. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Running, Role: LEADER
I20240426 09:54:29.643705 9736 consensus_queue.cc:237] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } }
I20240426 09:54:29.649461 9315 catalog_manager.cc:5484] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 reported cstate change: term changed from 2 to 3, leader changed from a51b16892b1f45a6933030491fcc05aa (127.8.141.129) to 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131). New cstate: current_term: 3 leader_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:30.077875 9778 raft_consensus.cc:979] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa: : Instructing follower 88b95908620241b7b6992d56d159a653 to start an election
I20240426 09:54:30.078346 9778 raft_consensus.cc:1067] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Signalling peer 88b95908620241b7b6992d56d159a653 to start an election
I20240426 09:54:30.080071 9495 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99873d472db84e08889ea87be1b4d6df"
dest_uuid: "88b95908620241b7b6992d56d159a653"
from {username='slave'} at 127.0.0.1:48370
I20240426 09:54:30.080585 9495 raft_consensus.cc:482] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:30.080873 9495 raft_consensus.cc:3037] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:30.080933 9419 raft_consensus.cc:1261] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Refusing update from remote peer 7b4e36cec98d4a4eb5e1f6c5ab762f19: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:54:30.082228 9736 consensus_queue.cc:1035] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:30.090281 9733 raft_consensus.cc:979] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa: : Instructing follower 88b95908620241b7b6992d56d159a653 to start an election
I20240426 09:54:30.090673 9733 raft_consensus.cc:1067] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 3 LEADER]: Signalling peer 88b95908620241b7b6992d56d159a653 to start an election
I20240426 09:54:30.090448 9495 raft_consensus.cc:504] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:30.092316 9494 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e"
dest_uuid: "88b95908620241b7b6992d56d159a653"
from {username='slave'} at 127.0.0.1:48370
I20240426 09:54:30.092769 9494 raft_consensus.cc:482] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:30.093078 9494 raft_consensus.cc:3037] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:54:30.093497 9495 leader_election.cc:288] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 3 election: Requested vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:30.095178 9563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99873d472db84e08889ea87be1b4d6df" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:30.095633 9563 raft_consensus.cc:3037] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:30.099387 9778 raft_consensus.cc:979] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa: : Instructing follower 7b4e36cec98d4a4eb5e1f6c5ab762f19 to start an election
I20240426 09:54:30.099838 9778 raft_consensus.cc:1067] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 4 LEADER]: Signalling peer 7b4e36cec98d4a4eb5e1f6c5ab762f19 to start an election
I20240426 09:54:30.102277 9568 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c"
dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
from {username='slave'} at 127.0.0.1:36752
I20240426 09:54:30.102887 9568 raft_consensus.cc:482] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:54:30.103263 9568 raft_consensus.cc:3037] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:54:30.103343 9563 raft_consensus.cc:2454] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 3.
I20240426 09:54:30.107875 9461 leader_election.cc:302] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653; no voters:
I20240426 09:54:30.107702 9494 raft_consensus.cc:504] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:30.109200 9776 raft_consensus.cc:2781] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:54:30.109572 9495 raft_consensus.cc:1261] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Refusing update from remote peer 7b4e36cec98d4a4eb5e1f6c5ab762f19: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:54:30.110208 9568 raft_consensus.cc:504] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:30.112761 9568 leader_election.cc:288] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 5 election: Requested vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215)
I20240426 09:54:30.113798 9494 leader_election.cc:288] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 4 election: Requested vote from peers a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159), 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131:40209)
I20240426 09:54:30.114755 9492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "88b95908620241b7b6992d56d159a653"
I20240426 09:54:30.114897 9420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99873d472db84e08889ea87be1b4d6df" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
I20240426 09:54:30.109611 9776 raft_consensus.cc:686] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 3 LEADER]: Becoming Leader. State: Replica: 88b95908620241b7b6992d56d159a653, State: Running, Role: LEADER
I20240426 09:54:30.115463 9736 consensus_queue.cc:1035] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:30.116205 9418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
I20240426 09:54:30.116767 9418 raft_consensus.cc:3032] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:54:30.116401 9776 consensus_queue.cc:237] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:30.117055 9418 raft_consensus.cc:729] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:30.115470 9420 raft_consensus.cc:3032] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:54:30.117660 9420 raft_consensus.cc:729] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:30.117827 9418 consensus_queue.cc:260] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:30.118474 9420 consensus_queue.cc:260] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:30.118947 9417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbd3b59d2fc41d49d30fc63daf3d20c" candidate_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "a51b16892b1f45a6933030491fcc05aa"
I20240426 09:54:30.119313 9418 raft_consensus.cc:3037] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:54:30.119894 9420 raft_consensus.cc:3037] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:54:30.120895 9417 raft_consensus.cc:3032] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:54:30.121232 9417 raft_consensus.cc:729] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a51b16892b1f45a6933030491fcc05aa, State: Running, Role: LEADER
I20240426 09:54:30.121989 9417 consensus_queue.cc:260] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:30.115247 9492 raft_consensus.cc:3037] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:54:30.123286 9417 raft_consensus.cc:3037] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:54:30.126704 9418 raft_consensus.cc:2454] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 4.
I20240426 09:54:30.130936 9492 raft_consensus.cc:2454] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in term 5.
I20240426 09:54:30.132419 9532 leader_election.cc:302] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b4e36cec98d4a4eb5e1f6c5ab762f19, 88b95908620241b7b6992d56d159a653; no voters:
I20240426 09:54:30.133445 9736 raft_consensus.cc:2781] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:54:30.133695 9461 leader_election.cc:302] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b95908620241b7b6992d56d159a653, a51b16892b1f45a6933030491fcc05aa; no voters:
I20240426 09:54:30.133988 9736 raft_consensus.cc:686] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 5 LEADER]: Becoming Leader. State: Replica: 7b4e36cec98d4a4eb5e1f6c5ab762f19, State: Running, Role: LEADER
I20240426 09:54:30.134702 9775 raft_consensus.cc:2781] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:54:30.135041 9736 consensus_queue.cc:237] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:30.134886 9417 raft_consensus.cc:2454] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b4e36cec98d4a4eb5e1f6c5ab762f19 in term 5.
I20240426 09:54:30.137756 9775 raft_consensus.cc:686] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 4 LEADER]: Becoming Leader. State: Replica: 88b95908620241b7b6992d56d159a653, State: Running, Role: LEADER
I20240426 09:54:30.138556 9775 consensus_queue.cc:237] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } }
I20240426 09:54:30.143649 9316 catalog_manager.cc:5484] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 reported cstate change: term changed from 4 to 5, leader changed from a51b16892b1f45a6933030491fcc05aa (127.8.141.129) to 7b4e36cec98d4a4eb5e1f6c5ab762f19 (127.8.141.131). New cstate: current_term: 5 leader_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:30.144636 9563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75da9d0d79d24604a4bb97685008c10e" candidate_uuid: "88b95908620241b7b6992d56d159a653" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19"
I20240426 09:54:30.145285 9563 raft_consensus.cc:3037] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:54:30.149971 9420 raft_consensus.cc:2454] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 3.
I20240426 09:54:30.158422 9563 raft_consensus.cc:2454] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b95908620241b7b6992d56d159a653 in term 4.
I20240426 09:54:30.161855 9316 catalog_manager.cc:5484] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 reported cstate change: term changed from 2 to 3, leader changed from a51b16892b1f45a6933030491fcc05aa (127.8.141.129) to 88b95908620241b7b6992d56d159a653 (127.8.141.130). New cstate: current_term: 3 leader_uuid: "88b95908620241b7b6992d56d159a653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:30.172029 9316 catalog_manager.cc:5484] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 reported cstate change: term changed from 3 to 4, leader changed from a51b16892b1f45a6933030491fcc05aa (127.8.141.129) to 88b95908620241b7b6992d56d159a653 (127.8.141.130). New cstate: current_term: 4 leader_uuid: "88b95908620241b7b6992d56d159a653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:30.535719 9563 raft_consensus.cc:1261] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Refusing update from remote peer 88b95908620241b7b6992d56d159a653: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:54:30.536803 9776 consensus_queue.cc:1035] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:30.545557 9419 raft_consensus.cc:1261] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Refusing update from remote peer 88b95908620241b7b6992d56d159a653: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:54:30.546643 9776 consensus_queue.cc:1035] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:30.582639 9419 raft_consensus.cc:1261] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 5 FOLLOWER]: Refusing update from remote peer 7b4e36cec98d4a4eb5e1f6c5ab762f19: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240426 09:54:30.583596 9736 consensus_queue.cc:1035] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:54:30.592484 9494 raft_consensus.cc:1261] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 5 FOLLOWER]: Refusing update from remote peer 7b4e36cec98d4a4eb5e1f6c5ab762f19: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240426 09:54:30.594516 9736 consensus_queue.cc:1035] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b95908620241b7b6992d56d159a653" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:54:30.665432 9419 raft_consensus.cc:1261] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 4 FOLLOWER]: Refusing update from remote peer 88b95908620241b7b6992d56d159a653: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:54:30.666344 9776 consensus_queue.cc:1035] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51b16892b1f45a6933030491fcc05aa" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:30.673358 9563 raft_consensus.cc:1261] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 4 FOLLOWER]: Refusing update from remote peer 88b95908620241b7b6992d56d159a653: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:54:30.674464 9786 consensus_queue.cc:1035] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4e36cec98d4a4eb5e1f6c5ab762f19" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:30.853585 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:30.859637 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:30.859983 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240426 09:54:31.060190 9590 debug-util.cc:398] Leaking SignalData structure 0x7b0800201a00 after lost signal to thread 8761
W20240426 09:54:31.061034 9590 debug-util.cc:398] Leaking SignalData structure 0x7b08001fdd40 after lost signal to thread 9352
W20240426 09:54:31.061805 9590 debug-util.cc:398] Leaking SignalData structure 0x7b0800119960 after lost signal to thread 9444
W20240426 09:54:31.062498 9590 debug-util.cc:398] Leaking SignalData structure 0x7b080006f760 after lost signal to thread 9519
W20240426 09:54:31.063205 9590 debug-util.cc:398] Leaking SignalData structure 0x7b08000e3360 after lost signal to thread 9593
I20240426 09:54:31.860646 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:31.865895 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:31.866233 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:32.866935 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:32.872030 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:32.872385 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:33.873037 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:33.877975 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:33.878330 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:35.504067 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:35.509457 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:35.509835 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240426 09:54:36.272560 9590 debug-util.cc:398] Leaking SignalData structure 0x7b08001c5920 after lost signal to thread 8761
W20240426 09:54:36.273207 9590 debug-util.cc:398] Leaking SignalData structure 0x7b0800153a40 after lost signal to thread 9352
W20240426 09:54:36.273901 9590 debug-util.cc:398] Leaking SignalData structure 0x7b080015f3a0 after lost signal to thread 9444
W20240426 09:54:36.274536 9590 debug-util.cc:398] Leaking SignalData structure 0x7b080015ffe0 after lost signal to thread 9519
W20240426 09:54:36.275179 9590 debug-util.cc:398] Leaking SignalData structure 0x7b08000ae880 after lost signal to thread 9593
I20240426 09:54:36.510545 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:36.516613 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:36.516988 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:37.517709 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:37.545480 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:37.545982 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:38.546662 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:38.552043 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:38.552507 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:39.553287 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:54:39.558252 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 09:54:39.558596 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:54:40.580134 8758 auto_leader_rebalancer-test.cc:376] The leader distribution is
I20240426 09:54:40.580698 8758 tablet_server.cc:178] TabletServer@127.8.141.129:0 shutting down...
I20240426 09:54:40.606140 8758 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 09:54:40.606930 8758 tablet_replica.cc:330] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa: stopping tablet replica
I20240426 09:54:40.607641 8758 raft_consensus.cc:2229] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 5 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.608155 8758 raft_consensus.cc:2258] T 9bbd3b59d2fc41d49d30fc63daf3d20c P a51b16892b1f45a6933030491fcc05aa [term 5 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.609972 8758 tablet_replica.cc:330] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa: stopping tablet replica
I20240426 09:54:40.610437 8758 raft_consensus.cc:2229] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.610926 8758 raft_consensus.cc:2258] T 87f66b3dfc3d4b5abb5726efb3d225f1 P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.613202 8758 tablet_replica.cc:330] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa: stopping tablet replica
I20240426 09:54:40.613696 8758 raft_consensus.cc:2229] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.614120 8758 raft_consensus.cc:2258] T 75da9d0d79d24604a4bb97685008c10e P a51b16892b1f45a6933030491fcc05aa [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.615607 8758 tablet_replica.cc:330] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa: stopping tablet replica
I20240426 09:54:40.616041 8758 raft_consensus.cc:2229] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.616423 8758 raft_consensus.cc:2258] T 99873d472db84e08889ea87be1b4d6df P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.617925 8758 tablet_replica.cc:330] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa: stopping tablet replica
I20240426 09:54:40.618368 8758 raft_consensus.cc:2229] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Raft consensus shutting down.
I20240426 09:54:40.619341 8758 raft_consensus.cc:2258] T 313a150ad860483a978ae5584f5554e0 P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.621315 8758 tablet_replica.cc:330] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa: stopping tablet replica
I20240426 09:54:40.621903 8758 raft_consensus.cc:2229] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 4 LEADER]: Raft consensus shutting down.
I20240426 09:54:40.622581 8758 raft_consensus.cc:2258] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P a51b16892b1f45a6933030491fcc05aa [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.624033 8758 tablet_replica.cc:330] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa: stopping tablet replica
I20240426 09:54:40.624437 8758 raft_consensus.cc:2229] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.624804 8758 raft_consensus.cc:2258] T 32407ee797ba4359a5f79513e69741ad P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.626353 8758 tablet_replica.cc:330] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa: stopping tablet replica
I20240426 09:54:40.626744 8758 raft_consensus.cc:2229] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.627138 8758 raft_consensus.cc:2258] T ff15116e7e7546b1b7b04c5b227e503e P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.628559 8758 tablet_replica.cc:330] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa: stopping tablet replica
I20240426 09:54:40.628947 8758 raft_consensus.cc:2229] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 3 LEADER]: Raft consensus shutting down.
I20240426 09:54:40.629597 8758 raft_consensus.cc:2258] T b1990a88d033491bbf9904c47e91a23d P a51b16892b1f45a6933030491fcc05aa [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.631060 8758 tablet_replica.cc:330] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa: stopping tablet replica
I20240426 09:54:40.631422 8758 raft_consensus.cc:2229] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [term 2 LEADER]: Raft consensus shutting down.
I20240426 09:54:40.632076 8758 raft_consensus.cc:2258] T b488843f1cb64132982e15fc409939fb P a51b16892b1f45a6933030491fcc05aa [term 2 FOLLOWER]: Raft consensus is shut down!
W20240426 09:54:40.665659 9461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.141.129:33159: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240426 09:54:40.668613 9461 consensus_peers.cc:485] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 -> Peer a51b16892b1f45a6933030491fcc05aa (127.8.141.129:33159): Couldn't send request to peer a51b16892b1f45a6933030491fcc05aa. Status: Network error: Client connection negotiation failed: client connection to 127.8.141.129:33159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:40.678566 8758 tablet_server.cc:194] TabletServer@127.8.141.129:0 shutdown complete.
I20240426 09:54:40.695209 8758 tablet_server.cc:178] TabletServer@127.8.141.130:0 shutting down...
I20240426 09:54:40.716744 8758 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 09:54:40.717352 8758 tablet_replica.cc:330] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653: stopping tablet replica
I20240426 09:54:40.718106 8758 raft_consensus.cc:2229] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.718540 8758 raft_consensus.cc:2258] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.720198 8758 tablet_replica.cc:330] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653: stopping tablet replica
I20240426 09:54:40.720628 8758 raft_consensus.cc:2229] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 4 LEADER]: Raft consensus shutting down.
I20240426 09:54:40.721362 8758 raft_consensus.cc:2258] T 75da9d0d79d24604a4bb97685008c10e P 88b95908620241b7b6992d56d159a653 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.722911 8758 tablet_replica.cc:330] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653: stopping tablet replica
I20240426 09:54:40.723326 8758 raft_consensus.cc:2229] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.723664 8758 raft_consensus.cc:2258] T 313a150ad860483a978ae5584f5554e0 P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.725118 8758 tablet_replica.cc:330] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653: stopping tablet replica
I20240426 09:54:40.725558 8758 raft_consensus.cc:2229] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.725935 8758 raft_consensus.cc:2258] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 88b95908620241b7b6992d56d159a653 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.727532 8758 tablet_replica.cc:330] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653: stopping tablet replica
I20240426 09:54:40.727967 8758 raft_consensus.cc:2229] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 3 LEADER]: Raft consensus shutting down.
I20240426 09:54:40.728592 8758 raft_consensus.cc:2258] T 99873d472db84e08889ea87be1b4d6df P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.730036 8758 tablet_replica.cc:330] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653: stopping tablet replica
I20240426 09:54:40.730444 8758 raft_consensus.cc:2229] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [term 2 LEADER]: Raft consensus shutting down.
I20240426 09:54:40.731154 8758 raft_consensus.cc:2258] T 32407ee797ba4359a5f79513e69741ad P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.732525 8758 tablet_replica.cc:330] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653: stopping tablet replica
I20240426 09:54:40.732923 8758 raft_consensus.cc:2229] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.733281 8758 raft_consensus.cc:2258] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 88b95908620241b7b6992d56d159a653 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.734926 8758 tablet_replica.cc:330] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653: stopping tablet replica
I20240426 09:54:40.735312 8758 raft_consensus.cc:2229] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.735637 8758 raft_consensus.cc:2258] T ff15116e7e7546b1b7b04c5b227e503e P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.737006 8758 tablet_replica.cc:330] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653: stopping tablet replica
I20240426 09:54:40.737421 8758 raft_consensus.cc:2229] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.737810 8758 raft_consensus.cc:2258] T b1990a88d033491bbf9904c47e91a23d P 88b95908620241b7b6992d56d159a653 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.739267 8758 tablet_replica.cc:330] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653: stopping tablet replica
I20240426 09:54:40.739637 8758 raft_consensus.cc:2229] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.739981 8758 raft_consensus.cc:2258] T b488843f1cb64132982e15fc409939fb P 88b95908620241b7b6992d56d159a653 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.766597 8758 tablet_server.cc:194] TabletServer@127.8.141.130:0 shutdown complete.
W20240426 09:54:40.772652 9532 consensus_peers.cc:485] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 -> Peer 88b95908620241b7b6992d56d159a653 (127.8.141.130:43215): Couldn't send request to peer 88b95908620241b7b6992d56d159a653. Status: Network error: Client connection negotiation failed: client connection to 127.8.141.130:43215: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:40.785162 8758 tablet_server.cc:178] TabletServer@127.8.141.131:0 shutting down...
I20240426 09:54:40.808633 8758 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 09:54:40.809321 8758 tablet_replica.cc:330] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19: stopping tablet replica
I20240426 09:54:40.810055 8758 raft_consensus.cc:2229] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 LEADER]: Raft consensus shutting down.
I20240426 09:54:40.811033 8758 raft_consensus.cc:2258] T ff15116e7e7546b1b7b04c5b227e503e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.812636 8758 tablet_replica.cc:330] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: stopping tablet replica
I20240426 09:54:40.813077 8758 raft_consensus.cc:2229] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.813418 8758 raft_consensus.cc:2258] T 313a150ad860483a978ae5584f5554e0 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.814846 8758 tablet_replica.cc:330] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19: stopping tablet replica
I20240426 09:54:40.815280 8758 raft_consensus.cc:2229] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.815635 8758 raft_consensus.cc:2258] T 75da9d0d79d24604a4bb97685008c10e P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.817019 8758 tablet_replica.cc:330] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19: stopping tablet replica
I20240426 09:54:40.817432 8758 raft_consensus.cc:2229] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.817770 8758 raft_consensus.cc:2258] T 32407ee797ba4359a5f79513e69741ad P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.819319 8758 tablet_replica.cc:330] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: stopping tablet replica
I20240426 09:54:40.819710 8758 raft_consensus.cc:2229] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.820065 8758 raft_consensus.cc:2258] T 3db2ebff0e144fc1a1a5144e9c6f7e60 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.821450 8758 tablet_replica.cc:330] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19: stopping tablet replica
I20240426 09:54:40.821892 8758 raft_consensus.cc:2229] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.822233 8758 raft_consensus.cc:2258] T 99873d472db84e08889ea87be1b4d6df P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.823594 8758 tablet_replica.cc:330] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19: stopping tablet replica
I20240426 09:54:40.824015 8758 raft_consensus.cc:2229] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 5 LEADER]: Raft consensus shutting down.
I20240426 09:54:40.824667 8758 raft_consensus.cc:2258] T 9bbd3b59d2fc41d49d30fc63daf3d20c P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.826169 8758 tablet_replica.cc:330] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19: stopping tablet replica
I20240426 09:54:40.826575 8758 raft_consensus.cc:2229] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.826948 8758 raft_consensus.cc:2258] T b488843f1cb64132982e15fc409939fb P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.828341 8758 tablet_replica.cc:330] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19: stopping tablet replica
I20240426 09:54:40.828724 8758 raft_consensus.cc:2229] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:40.829062 8758 raft_consensus.cc:2258] T b1990a88d033491bbf9904c47e91a23d P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.830607 8758 tablet_replica.cc:330] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19: stopping tablet replica
I20240426 09:54:40.831012 8758 raft_consensus.cc:2229] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 LEADER]: Raft consensus shutting down.
I20240426 09:54:40.831656 8758 raft_consensus.cc:2258] T 87f66b3dfc3d4b5abb5726efb3d225f1 P 7b4e36cec98d4a4eb5e1f6c5ab762f19 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.876471 8758 tablet_server.cc:194] TabletServer@127.8.141.131:0 shutdown complete.
I20240426 09:54:40.892758 8758 master.cc:537] Master@127.8.141.190:35665 shutting down...
I20240426 09:54:40.912076 8758 raft_consensus.cc:2229] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [term 1 LEADER]: Raft consensus shutting down.
I20240426 09:54:40.912537 8758 raft_consensus.cc:2258] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:40.912806 8758 tablet_replica.cc:330] T 00000000000000000000000000000000 P aa415a75df1d436aac70627c304b84aa: stopping tablet replica
I20240426 09:54:40.930336 8758 master.cc:552] Master@127.8.141.190:35665 shutdown complete.
7b4e36cec98d4a4eb5e1f6c5ab762f19 1
88b95908620241b7b6992d56d159a653 4
a51b16892b1f45a6933030491fcc05aa 5
7b4e36cec98d4a4eb5e1f6c5ab762f19 3
88b95908620241b7b6992d56d159a653 4
a51b16892b1f45a6933030491fcc05aa 3
7b4e36cec98d4a4eb5e1f6c5ab762f19 1
88b95908620241b7b6992d56d159a653 1
a51b16892b1f45a6933030491fcc05aa 8
7b4e36cec98d4a4eb5e1f6c5ab762f19 3
88b95908620241b7b6992d56d159a653 3
a51b16892b1f45a6933030491fcc05aa 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29341 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20240426 09:54:40.966166 8758 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.8.141.190:45863
I20240426 09:54:40.967185 8758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:54:40.972453 9814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:54:40.973066 9815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:54:40.973372 9817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:54:40.973857 8758 server_base.cc:1034] running on GCE node
I20240426 09:54:40.974669 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 09:54:40.974879 8758 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 09:54:40.975005 8758 hybrid_clock.cc:648] HybridClock initialized: now 1714125280974994 us; error 0 us; skew 500 ppm
I20240426 09:54:40.975456 8758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:54:40.980731 8758 webserver.cc:460] Webserver started at http://127.8.141.190:46829/ using document root <none> and password file <none>
I20240426 09:54:40.981141 8758 fs_manager.cc:359] Metadata directory not provided
I20240426 09:54:40.981287 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:54:40.981505 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:54:40.982553 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/master-0-root/instance:
uuid: "d0e04ed59b99401d8e95cbeab83582c3"
format_stamp: "Formatted at 2024-04-26 09:54:40 on dist-test-slave-5txv"
I20240426 09:54:40.986479 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240426 09:54:40.989401 9822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:40.990046 8758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240426 09:54:40.990270 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/master-0-root
uuid: "d0e04ed59b99401d8e95cbeab83582c3"
format_stamp: "Formatted at 2024-04-26 09:54:40 on dist-test-slave-5txv"
I20240426 09:54:40.990490 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:54:41.004804 8758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 09:54:41.005833 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:54:41.039510 8758 rpc_server.cc:307] RPC server started. Bound to: 127.8.141.190:45863
I20240426 09:54:41.039611 9873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.141.190:45863 every 8 connection(s)
I20240426 09:54:41.043021 9874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.052143 9874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3: Bootstrap starting.
I20240426 09:54:41.055913 9874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:41.059492 9874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3: No bootstrap required, opened a new log
I20240426 09:54:41.061277 9874 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e04ed59b99401d8e95cbeab83582c3" member_type: VOTER }
I20240426 09:54:41.061736 9874 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:41.061946 9874 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e04ed59b99401d8e95cbeab83582c3, State: Initialized, Role: FOLLOWER
I20240426 09:54:41.062429 9874 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e04ed59b99401d8e95cbeab83582c3" member_type: VOTER }
I20240426 09:54:41.062881 9874 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240426 09:54:41.063081 9874 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240426 09:54:41.063338 9874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:41.066850 9874 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e04ed59b99401d8e95cbeab83582c3" member_type: VOTER }
I20240426 09:54:41.067310 9874 leader_election.cc:302] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0e04ed59b99401d8e95cbeab83582c3; no voters:
I20240426 09:54:41.068250 9874 leader_election.cc:288] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240426 09:54:41.068560 9877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:41.069633 9877 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [term 1 LEADER]: Becoming Leader. State: Replica: d0e04ed59b99401d8e95cbeab83582c3, State: Running, Role: LEADER
I20240426 09:54:41.070231 9877 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e04ed59b99401d8e95cbeab83582c3" member_type: VOTER }
I20240426 09:54:41.070798 9874 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [sys.catalog]: configured and running, proceeding with master startup.
I20240426 09:54:41.074517 9878 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0e04ed59b99401d8e95cbeab83582c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e04ed59b99401d8e95cbeab83582c3" member_type: VOTER } }
I20240426 09:54:41.074648 9879 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0e04ed59b99401d8e95cbeab83582c3. Latest consensus state: current_term: 1 leader_uuid: "d0e04ed59b99401d8e95cbeab83582c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e04ed59b99401d8e95cbeab83582c3" member_type: VOTER } }
I20240426 09:54:41.075139 9878 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [sys.catalog]: This master's current role is: LEADER
I20240426 09:54:41.075356 9879 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3 [sys.catalog]: This master's current role is: LEADER
I20240426 09:54:41.079712 9884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240426 09:54:41.086139 9884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240426 09:54:41.087003 8758 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240426 09:54:41.093556 9884 catalog_manager.cc:1347] Generated new cluster ID: a9107c6a5213406c8a2209b853fc276d
I20240426 09:54:41.093776 9884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240426 09:54:41.115537 9884 catalog_manager.cc:1370] Generated new certificate authority record
I20240426 09:54:41.116601 9884 catalog_manager.cc:1504] Loading token signing keys...
I20240426 09:54:41.129202 9884 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3: Generated new TSK 0
I20240426 09:54:41.129676 9884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240426 09:54:41.152807 8758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:54:41.158325 9895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:54:41.159685 9896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:54:41.160758 9898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:54:41.162040 8758 server_base.cc:1034] running on GCE node
I20240426 09:54:41.162909 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 09:54:41.163102 8758 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 09:54:41.163250 8758 hybrid_clock.cc:648] HybridClock initialized: now 1714125281163233 us; error 0 us; skew 500 ppm
I20240426 09:54:41.163763 8758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:54:41.165951 8758 webserver.cc:460] Webserver started at http://127.8.141.129:33057/ using document root <none> and password file <none>
I20240426 09:54:41.166390 8758 fs_manager.cc:359] Metadata directory not provided
I20240426 09:54:41.166563 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:54:41.166847 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:54:41.167938 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-0-root/instance:
uuid: "f12521b6d51242629cf4297864df00b9"
format_stamp: "Formatted at 2024-04-26 09:54:41 on dist-test-slave-5txv"
I20240426 09:54:41.172194 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240426 09:54:41.175150 9903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:41.175797 8758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240426 09:54:41.176038 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-0-root
uuid: "f12521b6d51242629cf4297864df00b9"
format_stamp: "Formatted at 2024-04-26 09:54:41 on dist-test-slave-5txv"
I20240426 09:54:41.176276 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:54:41.193150 8758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 09:54:41.194132 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:54:41.195441 8758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:54:41.197393 8758 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:54:41.197558 8758 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:41.197733 8758 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:54:41.197875 8758 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:41.233048 8758 rpc_server.cc:307] RPC server started. Bound to: 127.8.141.129:38113
I20240426 09:54:41.233145 9965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.141.129:38113 every 8 connection(s)
I20240426 09:54:41.237072 8758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:54:41.244136 9970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:54:41.246850 9973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:54:41.247115 9971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:54:41.248030 8758 server_base.cc:1034] running on GCE node
I20240426 09:54:41.249012 9966 heartbeater.cc:347] Connected to a master server at 127.8.141.190:45863
I20240426 09:54:41.249183 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240426 09:54:41.249305 9966 heartbeater.cc:464] Registering TS with master...
W20240426 09:54:41.249429 8758 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 09:54:41.249620 8758 hybrid_clock.cc:648] HybridClock initialized: now 1714125281249604 us; error 0 us; skew 500 ppm
I20240426 09:54:41.250034 9966 heartbeater.cc:511] Master 127.8.141.190:45863 requested a full tablet report, sending...
I20240426 09:54:41.250166 8758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:54:41.252022 9839 ts_manager.cc:194] Registered new tserver with Master: f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113)
I20240426 09:54:41.252879 8758 webserver.cc:460] Webserver started at http://127.8.141.130:41791/ using document root <none> and password file <none>
I20240426 09:54:41.253494 8758 fs_manager.cc:359] Metadata directory not provided
I20240426 09:54:41.253719 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:54:41.253996 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:54:41.254335 9839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50240
I20240426 09:54:41.255409 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-1-root/instance:
uuid: "826f03b2fb9c4b9483623b041a1bf80e"
format_stamp: "Formatted at 2024-04-26 09:54:41 on dist-test-slave-5txv"
I20240426 09:54:41.259541 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240426 09:54:41.262605 9978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:41.263250 8758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240426 09:54:41.263497 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-1-root
uuid: "826f03b2fb9c4b9483623b041a1bf80e"
format_stamp: "Formatted at 2024-04-26 09:54:41 on dist-test-slave-5txv"
I20240426 09:54:41.263738 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:54:41.295454 8758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 09:54:41.296463 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:54:41.297664 8758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:54:41.299897 8758 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:54:41.300099 8758 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:41.300323 8758 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:54:41.300478 8758 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:41.336715 8758 rpc_server.cc:307] RPC server started. Bound to: 127.8.141.130:44411
I20240426 09:54:41.336803 10040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.141.130:44411 every 8 connection(s)
I20240426 09:54:41.340979 8758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:54:41.347697 10044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:54:41.348439 10045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:54:41.350530 8758 server_base.cc:1034] running on GCE node
W20240426 09:54:41.351338 10047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:54:41.351832 10041 heartbeater.cc:347] Connected to a master server at 127.8.141.190:45863
I20240426 09:54:41.352147 10041 heartbeater.cc:464] Registering TS with master...
I20240426 09:54:41.352164 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 09:54:41.352461 8758 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 09:54:41.352610 8758 hybrid_clock.cc:648] HybridClock initialized: now 1714125281352597 us; error 0 us; skew 500 ppm
I20240426 09:54:41.352897 10041 heartbeater.cc:511] Master 127.8.141.190:45863 requested a full tablet report, sending...
I20240426 09:54:41.353161 8758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:54:41.354753 9839 ts_manager.cc:194] Registered new tserver with Master: 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:41.355757 8758 webserver.cc:460] Webserver started at http://127.8.141.131:33285/ using document root <none> and password file <none>
I20240426 09:54:41.356380 8758 fs_manager.cc:359] Metadata directory not provided
I20240426 09:54:41.356527 9839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50252
I20240426 09:54:41.356612 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:54:41.357028 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:54:41.358142 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-2-root/instance:
uuid: "d3a480e831d649139ff016fb68aee3ee"
format_stamp: "Formatted at 2024-04-26 09:54:41 on dist-test-slave-5txv"
I20240426 09:54:41.362051 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240426 09:54:41.364886 10052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:41.365545 8758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240426 09:54:41.365810 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-2-root
uuid: "d3a480e831d649139ff016fb68aee3ee"
format_stamp: "Formatted at 2024-04-26 09:54:41 on dist-test-slave-5txv"
I20240426 09:54:41.366060 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:54:41.379660 8758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 09:54:41.380664 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:54:41.381915 8758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:54:41.383901 8758 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:54:41.384086 8758 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:41.384294 8758 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:54:41.384446 8758 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:41.420421 8758 rpc_server.cc:307] RPC server started. Bound to: 127.8.141.131:34821
I20240426 09:54:41.420509 10114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.141.131:34821 every 8 connection(s)
I20240426 09:54:41.432665 10115 heartbeater.cc:347] Connected to a master server at 127.8.141.190:45863
I20240426 09:54:41.432991 10115 heartbeater.cc:464] Registering TS with master...
I20240426 09:54:41.433625 10115 heartbeater.cc:511] Master 127.8.141.190:45863 requested a full tablet report, sending...
I20240426 09:54:41.435261 9839 ts_manager.cc:194] Registered new tserver with Master: d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:41.435420 8758 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.011975033s
I20240426 09:54:41.436910 9839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50254
I20240426 09:54:41.438174 8758 test_util.cc:268] Using random seed: -461362740
I20240426 09:54:41.458632 9839 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50264:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240426 09:54:41.464329 9839 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240426 09:54:41.653371 9931 tablet_service.cc:1467] Processing CreateTablet for tablet c6ed47e1980f4a14a46cc61ea156eb53 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240426 09:54:41.654727 9931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ed47e1980f4a14a46cc61ea156eb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.655897 9930 tablet_service.cc:1467] Processing CreateTablet for tablet 9c2caa6ec04942588af2288b0ac9ef6b (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240426 09:54:41.657120 9930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c2caa6ec04942588af2288b0ac9ef6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.664263 9929 tablet_service.cc:1467] Processing CreateTablet for tablet 5fc982aface54033a68641d8d809fd3b (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240426 09:54:41.665493 9929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fc982aface54033a68641d8d809fd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.666219 10080 tablet_service.cc:1467] Processing CreateTablet for tablet c6ed47e1980f4a14a46cc61ea156eb53 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240426 09:54:41.667428 10080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ed47e1980f4a14a46cc61ea156eb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.668843 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfada00f42a411f813bfc35d6c9ec9d (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240426 09:54:41.670029 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfada00f42a411f813bfc35d6c9ec9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.669775 10079 tablet_service.cc:1467] Processing CreateTablet for tablet 9c2caa6ec04942588af2288b0ac9ef6b (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240426 09:54:41.670953 10079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c2caa6ec04942588af2288b0ac9ef6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.671912 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 35fd22bc1cf64e38840dcfb850cfe090 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240426 09:54:41.673106 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35fd22bc1cf64e38840dcfb850cfe090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.672755 9928 tablet_service.cc:1467] Processing CreateTablet for tablet eb5d051be8c743a18cd550e145b6564e (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240426 09:54:41.673967 9928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5d051be8c743a18cd550e145b6564e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.675879 10078 tablet_service.cc:1467] Processing CreateTablet for tablet 5fc982aface54033a68641d8d809fd3b (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240426 09:54:41.676571 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 6151686b1a24447880e49b7d7395fcc1 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240426 09:54:41.677073 10078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fc982aface54033a68641d8d809fd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.677772 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6151686b1a24447880e49b7d7395fcc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.679005 10077 tablet_service.cc:1467] Processing CreateTablet for tablet eb5d051be8c743a18cd550e145b6564e (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240426 09:54:41.680189 10077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5d051be8c743a18cd550e145b6564e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.680347 9927 tablet_service.cc:1467] Processing CreateTablet for tablet bbe8813b366b4644872fbec0addf4443 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240426 09:54:41.681572 9927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe8813b366b4644872fbec0addf4443. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.681198 10006 tablet_service.cc:1467] Processing CreateTablet for tablet c6ed47e1980f4a14a46cc61ea156eb53 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240426 09:54:41.682497 10006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ed47e1980f4a14a46cc61ea156eb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.681150 10071 tablet_service.cc:1467] Processing CreateTablet for tablet 21e4aa688b194564bef9b4398f572caf (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240426 09:54:41.683473 10071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e4aa688b194564bef9b4398f572caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.683938 10076 tablet_service.cc:1467] Processing CreateTablet for tablet bbe8813b366b4644872fbec0addf4443 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240426 09:54:41.685115 10076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe8813b366b4644872fbec0addf4443. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.687885 10075 tablet_service.cc:1467] Processing CreateTablet for tablet 09b98bf14ce848eca1c7ca1840a516b1 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240426 09:54:41.688586 10005 tablet_service.cc:1467] Processing CreateTablet for tablet 9c2caa6ec04942588af2288b0ac9ef6b (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240426 09:54:41.689141 10075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b98bf14ce848eca1c7ca1840a516b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.688939 9925 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfada00f42a411f813bfc35d6c9ec9d (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240426 09:54:41.689780 10005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c2caa6ec04942588af2288b0ac9ef6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.690091 9925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfada00f42a411f813bfc35d6c9ec9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.687896 9926 tablet_service.cc:1467] Processing CreateTablet for tablet 09b98bf14ce848eca1c7ca1840a516b1 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240426 09:54:41.691767 9926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b98bf14ce848eca1c7ca1840a516b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.700863 10004 tablet_service.cc:1467] Processing CreateTablet for tablet 5fc982aface54033a68641d8d809fd3b (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240426 09:54:41.702142 10004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fc982aface54033a68641d8d809fd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.703899 10003 tablet_service.cc:1467] Processing CreateTablet for tablet eb5d051be8c743a18cd550e145b6564e (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240426 09:54:41.705092 10003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5d051be8c743a18cd550e145b6564e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.706529 9924 tablet_service.cc:1467] Processing CreateTablet for tablet 35fd22bc1cf64e38840dcfb850cfe090 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240426 09:54:41.707727 9924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35fd22bc1cf64e38840dcfb850cfe090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.707899 10002 tablet_service.cc:1467] Processing CreateTablet for tablet bbe8813b366b4644872fbec0addf4443 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240426 09:54:41.709144 10002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe8813b366b4644872fbec0addf4443. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.711347 9923 tablet_service.cc:1467] Processing CreateTablet for tablet 6151686b1a24447880e49b7d7395fcc1 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240426 09:54:41.712666 9923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6151686b1a24447880e49b7d7395fcc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.713092 10000 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfada00f42a411f813bfc35d6c9ec9d (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240426 09:54:41.714282 10000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfada00f42a411f813bfc35d6c9ec9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.715869 9999 tablet_service.cc:1467] Processing CreateTablet for tablet 35fd22bc1cf64e38840dcfb850cfe090 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240426 09:54:41.716385 9922 tablet_service.cc:1467] Processing CreateTablet for tablet 21e4aa688b194564bef9b4398f572caf (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240426 09:54:41.717080 9999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35fd22bc1cf64e38840dcfb850cfe090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.717686 9922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e4aa688b194564bef9b4398f572caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.719880 9998 tablet_service.cc:1467] Processing CreateTablet for tablet 6151686b1a24447880e49b7d7395fcc1 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240426 09:54:41.721035 9998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6151686b1a24447880e49b7d7395fcc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.723821 9997 tablet_service.cc:1467] Processing CreateTablet for tablet 21e4aa688b194564bef9b4398f572caf (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240426 09:54:41.724979 9997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e4aa688b194564bef9b4398f572caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.711869 10001 tablet_service.cc:1467] Processing CreateTablet for tablet 09b98bf14ce848eca1c7ca1840a516b1 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240426 09:54:41.734336 10001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b98bf14ce848eca1c7ca1840a516b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.756320 9924 tablet_service.cc:1467] Processing CreateTablet for tablet 50a3a93bd7c449cf86ef18473507bf75 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240426 09:54:41.757611 9924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a3a93bd7c449cf86ef18473507bf75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.772337 9930 tablet_service.cc:1467] Processing CreateTablet for tablet a3af5b1ab22b40b2abe93efd45bb7676 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240426 09:54:41.773636 9930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3af5b1ab22b40b2abe93efd45bb7676. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.809693 9997 tablet_service.cc:1467] Processing CreateTablet for tablet 50a3a93bd7c449cf86ef18473507bf75 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240426 09:54:41.839023 9997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a3a93bd7c449cf86ef18473507bf75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.860180 10131 tablet_bootstrap.cc:492] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:41.861289 10077 tablet_service.cc:1467] Processing CreateTablet for tablet 693cab52ca1f45cab4af9282a6614f90 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240426 09:54:41.862453 10077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693cab52ca1f45cab4af9282a6614f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.868561 9925 tablet_service.cc:1467] Processing CreateTablet for tablet bc3e15fcf67145f38946b62f0a5e9149 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240426 09:54:41.869768 9925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3e15fcf67145f38946b62f0a5e9149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.870038 10080 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0134eb2ced4f5c8e56412aaee1c225 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240426 09:54:41.871220 10080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0134eb2ced4f5c8e56412aaee1c225. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.871450 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 7b63eef1eb5e4fabb35fb66b0f5a41b4 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240426 09:54:41.873082 10131 tablet_bootstrap.cc:654] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:41.871450 10003 tablet_service.cc:1467] Processing CreateTablet for tablet 2e197425a8c74512b4dac9d31a3ce0a5 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240426 09:54:41.874032 10003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e197425a8c74512b4dac9d31a3ce0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.875680 10001 tablet_service.cc:1467] Processing CreateTablet for tablet bc3e15fcf67145f38946b62f0a5e9149 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240426 09:54:41.873041 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b63eef1eb5e4fabb35fb66b0f5a41b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.876876 10001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3e15fcf67145f38946b62f0a5e9149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.862461 10002 tablet_service.cc:1467] Processing CreateTablet for tablet a3af5b1ab22b40b2abe93efd45bb7676 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240426 09:54:41.883785 10002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3af5b1ab22b40b2abe93efd45bb7676. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.889981 10075 tablet_service.cc:1467] Processing CreateTablet for tablet 2e197425a8c74512b4dac9d31a3ce0a5 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240426 09:54:41.895485 10075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e197425a8c74512b4dac9d31a3ce0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.898941 10074 tablet_service.cc:1467] Processing CreateTablet for tablet ec8cf6b7af0543f3a118ab747864f366 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240426 09:54:41.863970 9922 tablet_service.cc:1467] Processing CreateTablet for tablet 2e197425a8c74512b4dac9d31a3ce0a5 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240426 09:54:41.905671 10080 tablet_service.cc:1467] Processing CreateTablet for tablet 1c21d6dedd1441bd977f09cdcf41d436 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240426 09:54:41.906544 9922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e197425a8c74512b4dac9d31a3ce0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.913617 10132 tablet_bootstrap.cc:492] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:41.900193 9999 tablet_service.cc:1467] Processing CreateTablet for tablet 693cab52ca1f45cab4af9282a6614f90 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240426 09:54:41.915107 9999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693cab52ca1f45cab4af9282a6614f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.907708 9931 tablet_service.cc:1467] Processing CreateTablet for tablet 693cab52ca1f45cab4af9282a6614f90 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240426 09:54:41.907677 10003 tablet_service.cc:1467] Processing CreateTablet for tablet 7b63eef1eb5e4fabb35fb66b0f5a41b4 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240426 09:54:41.908766 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8cf6b7af0543f3a118ab747864f366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.909348 10005 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0134eb2ced4f5c8e56412aaee1c225 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240426 09:54:41.909837 9927 tablet_service.cc:1467] Processing CreateTablet for tablet ec8cf6b7af0543f3a118ab747864f366 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240426 09:54:41.910672 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 50a3a93bd7c449cf86ef18473507bf75 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240426 09:54:41.891206 10130 tablet_bootstrap.cc:492] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:41.911414 9928 tablet_service.cc:1467] Processing CreateTablet for tablet 7b63eef1eb5e4fabb35fb66b0f5a41b4 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240426 09:54:41.911365 10000 tablet_service.cc:1467] Processing CreateTablet for tablet ec8cf6b7af0543f3a118ab747864f366 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240426 09:54:41.860217 10078 tablet_service.cc:1467] Processing CreateTablet for tablet bc3e15fcf67145f38946b62f0a5e9149 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240426 09:54:41.918562 10071 tablet_service.cc:1467] Processing CreateTablet for tablet 5c0fe10a2b3746f9909d3596580b6854 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240426 09:54:41.977144 10004 tablet_service.cc:1467] Processing CreateTablet for tablet 5c0fe10a2b3746f9909d3596580b6854 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240426 09:54:41.923557 10003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b63eef1eb5e4fabb35fb66b0f5a41b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.929554 9925 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0134eb2ced4f5c8e56412aaee1c225 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240426 09:54:41.979233 10005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0134eb2ced4f5c8e56412aaee1c225. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.919884 9931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693cab52ca1f45cab4af9282a6614f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.982863 9997 tablet_service.cc:1467] Processing CreateTablet for tablet 9da46d724daf4bbf9c49a393e3579156 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240426 09:54:41.983829 9927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8cf6b7af0543f3a118ab747864f366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.984320 10000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8cf6b7af0543f3a118ab747864f366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.918746 10132 tablet_bootstrap.cc:654] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:41.985184 9923 tablet_service.cc:1467] Processing CreateTablet for tablet 5c0fe10a2b3746f9909d3596580b6854 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240426 09:54:41.986285 9923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c0fe10a2b3746f9909d3596580b6854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.925491 10076 tablet_service.cc:1467] Processing CreateTablet for tablet 9da46d724daf4bbf9c49a393e3579156 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240426 09:54:41.987744 9928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b63eef1eb5e4fabb35fb66b0f5a41b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.990070 9997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da46d724daf4bbf9c49a393e3579156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.991832 10004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c0fe10a2b3746f9909d3596580b6854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.995328 9924 tablet_service.cc:1467] Processing CreateTablet for tablet 9da46d724daf4bbf9c49a393e3579156 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240426 09:54:41.997911 10130 tablet_bootstrap.cc:654] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:41.999086 9925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0134eb2ced4f5c8e56412aaee1c225. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.924605 10080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c21d6dedd1441bd977f09cdcf41d436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.930033 10006 tablet_service.cc:1467] Processing CreateTablet for tablet 472a3b5debc94412b63080f474ae4fed (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240426 09:54:42.028995 9999 tablet_service.cc:1467] Processing CreateTablet for tablet be50ac00c79946d89ecaf230084a401d (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240426 09:54:42.033828 10006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472a3b5debc94412b63080f474ae4fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.925863 10072 tablet_service.cc:1467] Processing CreateTablet for tablet a3af5b1ab22b40b2abe93efd45bb7676 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240426 09:54:42.035609 10001 tablet_service.cc:1467] Processing CreateTablet for tablet 69700f6b120b4f4ebc0b43b14166f4c9 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240426 09:54:42.036761 10001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69700f6b120b4f4ebc0b43b14166f4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.038796 9999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be50ac00c79946d89ecaf230084a401d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.043972 10131 tablet_bootstrap.cc:492] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:42.044458 10131 ts_tablet_manager.cc:1367] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.185s user 0.013s sys 0.019s
I20240426 09:54:42.047058 10131 raft_consensus.cc:348] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.048125 9924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da46d724daf4bbf9c49a393e3579156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.050329 9929 tablet_service.cc:1467] Processing CreateTablet for tablet be50ac00c79946d89ecaf230084a401d (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240426 09:54:42.052644 10131 raft_consensus.cc:374] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:42.058660 10131 raft_consensus.cc:729] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:42.059545 10131 consensus_queue.cc:260] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.063640 9922 tablet_service.cc:1467] Processing CreateTablet for tablet 69700f6b120b4f4ebc0b43b14166f4c9 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240426 09:54:42.064112 10071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c0fe10a2b3746f9909d3596580b6854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.064352 9929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be50ac00c79946d89ecaf230084a401d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.065210 9922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69700f6b120b4f4ebc0b43b14166f4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.067435 10076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da46d724daf4bbf9c49a393e3579156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.071471 10078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3e15fcf67145f38946b62f0a5e9149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.075044 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3af5b1ab22b40b2abe93efd45bb7676. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:41.959048 9926 tablet_service.cc:1467] Processing CreateTablet for tablet 1c21d6dedd1441bd977f09cdcf41d436 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240426 09:54:42.079524 9926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c21d6dedd1441bd977f09cdcf41d436. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 09:54:42.081310 9887 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 09:54:42.082743 10075 tablet_service.cc:1467] Processing CreateTablet for tablet 69700f6b120b4f4ebc0b43b14166f4c9 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240426 09:54:42.083884 10075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69700f6b120b4f4ebc0b43b14166f4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.088786 9928 tablet_service.cc:1467] Processing CreateTablet for tablet ac0131f6d1d54716ab3324104a18c9c3 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240426 09:54:42.089954 9928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac0131f6d1d54716ab3324104a18c9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.078320 10002 tablet_service.cc:1467] Processing CreateTablet for tablet 1b76245317c24e129e7f16cdb4d99007 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240426 09:54:41.931118 10079 tablet_service.cc:1467] Processing CreateTablet for tablet be50ac00c79946d89ecaf230084a401d (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240426 09:54:42.091405 10002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b76245317c24e129e7f16cdb4d99007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.091499 10079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be50ac00c79946d89ecaf230084a401d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.094457 10003 tablet_service.cc:1467] Processing CreateTablet for tablet ac0131f6d1d54716ab3324104a18c9c3 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240426 09:54:42.059463 9931 tablet_service.cc:1467] Processing CreateTablet for tablet 1b76245317c24e129e7f16cdb4d99007 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240426 09:54:42.095762 9931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b76245317c24e129e7f16cdb4d99007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.099006 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a3a93bd7c449cf86ef18473507bf75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.104135 9925 tablet_service.cc:1467] Processing CreateTablet for tablet 3c3eaa7da54b489e99a7451385057422 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240426 09:54:42.105240 9925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c3eaa7da54b489e99a7451385057422. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.106894 9922 tablet_service.cc:1467] Processing CreateTablet for tablet 5e80ebfd190643a99ef02b6ea2876426 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240426 09:54:42.113533 9922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e80ebfd190643a99ef02b6ea2876426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.095706 10003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac0131f6d1d54716ab3324104a18c9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.113858 10001 tablet_service.cc:1467] Processing CreateTablet for tablet 5e80ebfd190643a99ef02b6ea2876426 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240426 09:54:41.921972 9998 tablet_service.cc:1467] Processing CreateTablet for tablet 1c21d6dedd1441bd977f09cdcf41d436 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240426 09:54:42.114993 10001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e80ebfd190643a99ef02b6ea2876426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.031391 9930 tablet_service.cc:1467] Processing CreateTablet for tablet 472a3b5debc94412b63080f474ae4fed (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240426 09:54:42.104139 10130 tablet_bootstrap.cc:492] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:42.131862 9927 tablet_service.cc:1467] Processing CreateTablet for tablet 3dc043e5f7b241afb7119ba8b06ed37f (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240426 09:54:42.132835 10072 tablet_service.cc:1467] Processing CreateTablet for tablet ac0131f6d1d54716ab3324104a18c9c3 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240426 09:54:42.136473 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac0131f6d1d54716ab3324104a18c9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.126032 10130 ts_tablet_manager.cc:1367] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.235s user 0.011s sys 0.005s
I20240426 09:54:42.110600 9997 tablet_service.cc:1467] Processing CreateTablet for tablet 3c3eaa7da54b489e99a7451385057422 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240426 09:54:42.133402 9927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dc043e5f7b241afb7119ba8b06ed37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.145829 10130 raft_consensus.cc:348] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.146538 10130 raft_consensus.cc:374] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:42.147145 10005 tablet_service.cc:1467] Processing CreateTablet for tablet 3dc043e5f7b241afb7119ba8b06ed37f (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240426 09:54:42.147732 9998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c21d6dedd1441bd977f09cdcf41d436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.151460 10078 tablet_service.cc:1467] Processing CreateTablet for tablet 1b76245317c24e129e7f16cdb4d99007 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240426 09:54:42.154150 10132 tablet_bootstrap.cc:492] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:42.144898 10006 tablet_service.cc:1467] Processing CreateTablet for tablet cf8634deb0a5425db9c47915d49386d5 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240426 09:54:42.158111 10132 ts_tablet_manager.cc:1367] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.245s user 0.031s sys 0.022s
I20240426 09:54:42.155933 10130 raft_consensus.cc:729] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:42.170219 10077 tablet_service.cc:1467] Processing CreateTablet for tablet 472a3b5debc94412b63080f474ae4fed (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240426 09:54:42.167918 9930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472a3b5debc94412b63080f474ae4fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.174647 9929 tablet_service.cc:1467] Processing CreateTablet for tablet 2f96620cc62e4bf1beefad6b59ca3968 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240426 09:54:42.174849 9928 tablet_service.cc:1467] Processing CreateTablet for tablet cf8634deb0a5425db9c47915d49386d5 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240426 09:54:42.175813 9929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f96620cc62e4bf1beefad6b59ca3968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.176774 9928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf8634deb0a5425db9c47915d49386d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.174669 10132 raft_consensus.cc:348] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.177500 10132 raft_consensus.cc:374] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:42.177821 10132 raft_consensus.cc:729] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:42.178248 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 3c3eaa7da54b489e99a7451385057422 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240426 09:54:42.178625 10132 consensus_queue.cc:260] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.184357 10115 heartbeater.cc:503] Master 127.8.141.190:45863 was elected leader, sending a full tablet report...
I20240426 09:54:42.192353 9999 tablet_service.cc:1467] Processing CreateTablet for tablet 2f96620cc62e4bf1beefad6b59ca3968 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240426 09:54:42.195307 10131 ts_tablet_manager.cc:1398] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.151s user 0.018s sys 0.013s
I20240426 09:54:42.197718 10075 tablet_service.cc:1467] Processing CreateTablet for tablet 5e80ebfd190643a99ef02b6ea2876426 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240426 09:54:42.203367 10078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b76245317c24e129e7f16cdb4d99007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.205516 10131 tablet_bootstrap.cc:492] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:42.206105 10003 tablet_service.cc:1467] Processing CreateTablet for tablet 28378f040d6642db8ba229c32bfed21a (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240426 09:54:42.207052 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c3eaa7da54b489e99a7451385057422. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.217028 10071 tablet_service.cc:1467] Processing CreateTablet for tablet cf8634deb0a5425db9c47915d49386d5 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240426 09:54:42.219020 10076 tablet_service.cc:1467] Processing CreateTablet for tablet 2f96620cc62e4bf1beefad6b59ca3968 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240426 09:54:42.219372 9926 tablet_service.cc:1467] Processing CreateTablet for tablet 28378f040d6642db8ba229c32bfed21a (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240426 09:54:42.222921 10077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472a3b5debc94412b63080f474ae4fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.214365 10130 consensus_queue.cc:260] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.213194 9997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c3eaa7da54b489e99a7451385057422. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.228276 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 3dc043e5f7b241afb7119ba8b06ed37f (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240426 09:54:42.229641 10004 tablet_service.cc:1467] Processing CreateTablet for tablet 72a6c52e0c924ba3b3fca5f9586814a7 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240426 09:54:42.235599 10132 ts_tablet_manager.cc:1398] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.077s user 0.007s sys 0.003s
I20240426 09:54:42.235759 10005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dc043e5f7b241afb7119ba8b06ed37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.239225 10071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf8634deb0a5425db9c47915d49386d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.243018 10006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf8634deb0a5425db9c47915d49386d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.243286 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dc043e5f7b241afb7119ba8b06ed37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.246409 10041 heartbeater.cc:503] Master 127.8.141.190:45863 was elected leader, sending a full tablet report...
I20240426 09:54:42.247938 10004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a6c52e0c924ba3b3fca5f9586814a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.248099 10080 tablet_service.cc:1467] Processing CreateTablet for tablet 28378f040d6642db8ba229c32bfed21a (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240426 09:54:42.249238 10080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28378f040d6642db8ba229c32bfed21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.251006 9999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f96620cc62e4bf1beefad6b59ca3968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.230790 10000 tablet_service.cc:1467] Processing CreateTablet for tablet cfa61a21874f49688b31ddd60646a658 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240426 09:54:42.252068 10000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfa61a21874f49688b31ddd60646a658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.253126 10132 tablet_bootstrap.cc:492] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:42.253607 10079 tablet_service.cc:1467] Processing CreateTablet for tablet cfa61a21874f49688b31ddd60646a658 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240426 09:54:42.256584 9966 heartbeater.cc:503] Master 127.8.141.190:45863 was elected leader, sending a full tablet report...
I20240426 09:54:42.259483 10132 tablet_bootstrap.cc:654] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:42.254736 10079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfa61a21874f49688b31ddd60646a658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.275615 10001 tablet_service.cc:1467] Processing CreateTablet for tablet c75e7dd8bbe64a3eb2174cd4a218d51c (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240426 09:54:42.276064 9923 tablet_service.cc:1467] Processing CreateTablet for tablet 72a6c52e0c924ba3b3fca5f9586814a7 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240426 09:54:42.261971 10137 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:42.277232 9923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a6c52e0c924ba3b3fca5f9586814a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.276903 10137 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.277774 10076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f96620cc62e4bf1beefad6b59ca3968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.265017 10002 tablet_service.cc:1467] Processing CreateTablet for tablet 007f1451130c4dfcbd86145d56a6f67e (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240426 09:54:42.260324 10131 tablet_bootstrap.cc:654] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:42.279348 10137 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:42.262730 9924 tablet_service.cc:1467] Processing CreateTablet for tablet cfa61a21874f49688b31ddd60646a658 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240426 09:54:42.293116 10003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28378f040d6642db8ba229c32bfed21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.300804 9926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28378f040d6642db8ba229c32bfed21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.310899 9928 tablet_service.cc:1467] Processing CreateTablet for tablet 007f1451130c4dfcbd86145d56a6f67e (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240426 09:54:42.301837 9925 tablet_service.cc:1467] Processing CreateTablet for tablet c75e7dd8bbe64a3eb2174cd4a218d51c (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240426 09:54:42.303844 9931 tablet_service.cc:1467] Processing CreateTablet for tablet a202d889782b4735855ec95f82e22ac8 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240426 09:54:42.315037 10002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007f1451130c4dfcbd86145d56a6f67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.317008 10075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e80ebfd190643a99ef02b6ea2876426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.329491 10001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c75e7dd8bbe64a3eb2174cd4a218d51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.335729 9924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfa61a21874f49688b31ddd60646a658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.351480 9925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c75e7dd8bbe64a3eb2174cd4a218d51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.352098 9931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a202d889782b4735855ec95f82e22ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.356726 10136 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:42.358577 10136 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.360972 10136 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:42.379398 9930 tablet_service.cc:1467] Processing CreateTablet for tablet e117a0ad5cd34acea96de04537a80733 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240426 09:54:42.382735 10001 tablet_service.cc:1467] Processing CreateTablet for tablet a202d889782b4735855ec95f82e22ac8 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240426 09:54:42.387920 9928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007f1451130c4dfcbd86145d56a6f67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.394871 10001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a202d889782b4735855ec95f82e22ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.397122 10003 tablet_service.cc:1467] Processing CreateTablet for tablet e11b7521039c4feba17dc65ac6c819fb (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240426 09:54:42.398319 10003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11b7521039c4feba17dc65ac6c819fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.388765 10132 tablet_bootstrap.cc:492] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:42.399300 10132 ts_tablet_manager.cc:1367] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.146s user 0.012s sys 0.002s
I20240426 09:54:42.401801 10132 raft_consensus.cc:348] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.402477 10132 raft_consensus.cc:374] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:42.402747 10132 raft_consensus.cc:729] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:42.403797 10132 consensus_queue.cc:260] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.404845 10074 tablet_service.cc:1467] Processing CreateTablet for tablet c75e7dd8bbe64a3eb2174cd4a218d51c (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240426 09:54:42.406005 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c75e7dd8bbe64a3eb2174cd4a218d51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.389973 9927 tablet_service.cc:1467] Processing CreateTablet for tablet 5945f6dc42a14b56b9e933f0c1839de1 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240426 09:54:42.403060 9930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e117a0ad5cd34acea96de04537a80733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.410310 9997 tablet_service.cc:1467] Processing CreateTablet for tablet e117a0ad5cd34acea96de04537a80733 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240426 09:54:42.411741 10005 tablet_service.cc:1467] Processing CreateTablet for tablet 5945f6dc42a14b56b9e933f0c1839de1 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240426 09:54:42.389284 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 72a6c52e0c924ba3b3fca5f9586814a7 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240426 09:54:42.413450 10077 tablet_service.cc:1467] Processing CreateTablet for tablet 007f1451130c4dfcbd86145d56a6f67e (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240426 09:54:42.355854 9929 tablet_service.cc:1467] Processing CreateTablet for tablet e11b7521039c4feba17dc65ac6c819fb (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240426 09:54:42.421088 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a6c52e0c924ba3b3fca5f9586814a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.406023 10132 ts_tablet_manager.cc:1398] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240426 09:54:42.356810 9922 tablet_service.cc:1467] Processing CreateTablet for tablet 17e4aa133eb141699b9a5e2f0b4767d6 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240426 09:54:42.424309 9927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5945f6dc42a14b56b9e933f0c1839de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.433432 9929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11b7521039c4feba17dc65ac6c819fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.393618 9998 tablet_service.cc:1467] Processing CreateTablet for tablet 17e4aa133eb141699b9a5e2f0b4767d6 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240426 09:54:42.445447 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:42.447019 10077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007f1451130c4dfcbd86145d56a6f67e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 09:54:42.447402 9979 leader_election.cc:341] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:42.451284 10132 tablet_bootstrap.cc:492] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:42.458807 10071 tablet_service.cc:1467] Processing CreateTablet for tablet a202d889782b4735855ec95f82e22ac8 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240426 09:54:42.462182 10079 tablet_service.cc:1467] Processing CreateTablet for tablet e11b7521039c4feba17dc65ac6c819fb (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240426 09:54:42.467674 9922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17e4aa133eb141699b9a5e2f0b4767d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.480528 9931 tablet_service.cc:1467] Processing CreateTablet for tablet 51125fbde06b4a588fb6171dec56a605 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240426 09:54:42.482954 9931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51125fbde06b4a588fb6171dec56a605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.480528 9929 tablet_service.cc:1467] Processing CreateTablet for tablet 308d6ed4b68346be85caf2871916eab9 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240426 09:54:42.493079 10005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5945f6dc42a14b56b9e933f0c1839de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.495999 10001 tablet_service.cc:1467] Processing CreateTablet for tablet 308d6ed4b68346be85caf2871916eab9 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240426 09:54:42.498010 9997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e117a0ad5cd34acea96de04537a80733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.497912 10003 tablet_service.cc:1467] Processing CreateTablet for tablet 51125fbde06b4a588fb6171dec56a605 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240426 09:54:42.500413 9998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17e4aa133eb141699b9a5e2f0b4767d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.502246 10004 tablet_service.cc:1467] Processing CreateTablet for tablet ddc5aa33dc824544ade18161bf2800a6 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240426 09:54:42.503682 10003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51125fbde06b4a588fb6171dec56a605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.504225 9929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 308d6ed4b68346be85caf2871916eab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.506384 10001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 308d6ed4b68346be85caf2871916eab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.508678 10004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc5aa33dc824544ade18161bf2800a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.509068 10130 ts_tablet_manager.cc:1398] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.365s user 0.008s sys 0.009s
I20240426 09:54:42.511083 9999 tablet_service.cc:1467] Processing CreateTablet for tablet d9f79bd8994f4b229ca8da8638bb6d5a (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240426 09:54:42.512259 9999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f79bd8994f4b229ca8da8638bb6d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.512037 10002 tablet_service.cc:1467] Processing CreateTablet for tablet 60bebc132e4141bdbab199326986516e (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240426 09:54:42.513198 10002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60bebc132e4141bdbab199326986516e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.513978 10079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11b7521039c4feba17dc65ac6c819fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.519189 10071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a202d889782b4735855ec95f82e22ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.519781 10000 tablet_service.cc:1467] Processing CreateTablet for tablet ab7abcb2e80641eca676add7616cd966 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240426 09:54:42.520891 10000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7abcb2e80641eca676add7616cd966. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.512262 9926 tablet_service.cc:1467] Processing CreateTablet for tablet d9f79bd8994f4b229ca8da8638bb6d5a (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240426 09:54:42.525417 9926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f79bd8994f4b229ca8da8638bb6d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.528266 10132 tablet_bootstrap.cc:654] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
W20240426 09:54:42.529876 9823 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 103315us
W20240426 09:54:42.532238 9826 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 51740.3us
I20240426 09:54:42.532910 9927 tablet_service.cc:1467] Processing CreateTablet for tablet a7ef86a50a514e8ba09c76ca674b2100 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240426 09:54:42.534034 9927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ef86a50a514e8ba09c76ca674b2100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.534634 9924 tablet_service.cc:1467] Processing CreateTablet for tablet ddc5aa33dc824544ade18161bf2800a6 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240426 09:54:42.535809 9924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc5aa33dc824544ade18161bf2800a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.536664 9930 tablet_service.cc:1467] Processing CreateTablet for tablet 23f4904f98c241de901e223b5ab4725d (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240426 09:54:42.537874 9930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f4904f98c241de901e223b5ab4725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.542917 10130 tablet_bootstrap.cc:492] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:42.544586 10078 tablet_service.cc:1467] Processing CreateTablet for tablet 17e4aa133eb141699b9a5e2f0b4767d6 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240426 09:54:42.545732 10078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17e4aa133eb141699b9a5e2f0b4767d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.548869 9925 tablet_service.cc:1467] Processing CreateTablet for tablet 60bebc132e4141bdbab199326986516e (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240426 09:54:42.550071 9925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60bebc132e4141bdbab199326986516e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.536928 10006 tablet_service.cc:1467] Processing CreateTablet for tablet a7ef86a50a514e8ba09c76ca674b2100 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240426 09:54:42.551362 10006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ef86a50a514e8ba09c76ca674b2100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.532045 9923 tablet_service.cc:1467] Processing CreateTablet for tablet ab7abcb2e80641eca676add7616cd966 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240426 09:54:42.564229 9923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7abcb2e80641eca676add7616cd966. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.571750 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:42.572819 10130 tablet_bootstrap.cc:654] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
W20240426 09:54:42.573312 10055 leader_election.cc:341] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:42.575657 9928 tablet_service.cc:1467] Processing CreateTablet for tablet fa1ede5c9b614a0f8b7adb20a7cfbbb1 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240426 09:54:42.578567 10076 tablet_service.cc:1467] Processing CreateTablet for tablet e117a0ad5cd34acea96de04537a80733 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240426 09:54:42.579895 10076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e117a0ad5cd34acea96de04537a80733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.584408 10080 tablet_service.cc:1467] Processing CreateTablet for tablet 308d6ed4b68346be85caf2871916eab9 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240426 09:54:42.585593 10080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 308d6ed4b68346be85caf2871916eab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.591336 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:42.593060 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:42.593982 9930 tablet_service.cc:1467] Processing CreateTablet for tablet fb385fda44e94153ae3cd9ca21240939 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
W20240426 09:54:42.594605 10056 leader_election.cc:341] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:42.594962 9928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1ede5c9b614a0f8b7adb20a7cfbbb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.595063 10056 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:42.596115 10136 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240426 09:54:42.597141 9982 leader_election.cc:341] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:42.597574 9982 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
W20240426 09:54:42.598222 10042 tablet.cc:2369] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 09:54:42.598596 10137 raft_consensus.cc:2726] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:42.599491 9930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb385fda44e94153ae3cd9ca21240939. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.600997 9998 tablet_service.cc:1467] Processing CreateTablet for tablet 23f4904f98c241de901e223b5ab4725d (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240426 09:54:42.602154 9998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f4904f98c241de901e223b5ab4725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.609177 10000 tablet_service.cc:1467] Processing CreateTablet for tablet fa1ede5c9b614a0f8b7adb20a7cfbbb1 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240426 09:54:42.612524 10000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1ede5c9b614a0f8b7adb20a7cfbbb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.613776 10131 tablet_bootstrap.cc:492] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:42.614244 10131 ts_tablet_manager.cc:1367] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.409s user 0.008s sys 0.012s
I20240426 09:54:42.616869 10131 raft_consensus.cc:348] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.617543 10131 raft_consensus.cc:374] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:42.617851 10131 raft_consensus.cc:729] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:42.626178 9924 tablet_service.cc:1467] Processing CreateTablet for tablet 951e4c48fcb642e58ab79652d3f86261 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240426 09:54:42.627343 9924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 951e4c48fcb642e58ab79652d3f86261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.631243 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 5945f6dc42a14b56b9e933f0c1839de1 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240426 09:54:42.632413 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5945f6dc42a14b56b9e933f0c1839de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.632712 9997 tablet_service.cc:1467] Processing CreateTablet for tablet 951e4c48fcb642e58ab79652d3f86261 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240426 09:54:42.633946 9997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 951e4c48fcb642e58ab79652d3f86261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.643074 10079 tablet_service.cc:1467] Processing CreateTablet for tablet 51125fbde06b4a588fb6171dec56a605 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240426 09:54:42.644332 10079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51125fbde06b4a588fb6171dec56a605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.665890 10139 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:42.666357 10139 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.669564 10139 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:42.676841 10130 tablet_bootstrap.cc:492] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:42.677336 10130 ts_tablet_manager.cc:1367] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.135s user 0.016s sys 0.000s
I20240426 09:54:42.678400 9927 tablet_service.cc:1467] Processing CreateTablet for tablet 67dfdb63e28a460486dfe163f03bf820 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240426 09:54:42.679948 10130 raft_consensus.cc:348] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.682296 10130 raft_consensus.cc:374] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:42.682616 10130 raft_consensus.cc:729] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:42.683291 10130 consensus_queue.cc:260] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.684696 10003 tablet_service.cc:1467] Processing CreateTablet for tablet fb385fda44e94153ae3cd9ca21240939 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240426 09:54:42.685401 10077 tablet_service.cc:1467] Processing CreateTablet for tablet d9f79bd8994f4b229ca8da8638bb6d5a (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240426 09:54:42.686882 10072 tablet_service.cc:1467] Processing CreateTablet for tablet ddc5aa33dc824544ade18161bf2800a6 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240426 09:54:42.699376 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc5aa33dc824544ade18161bf2800a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.700480 9927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67dfdb63e28a460486dfe163f03bf820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.707262 10077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f79bd8994f4b229ca8da8638bb6d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.698618 9931 tablet_service.cc:1467] Processing CreateTablet for tablet 6fde852ed1264555af856522b4abc093 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240426 09:54:42.708226 9931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fde852ed1264555af856522b4abc093. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.709131 10137 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:42.709568 10137 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.709923 10005 tablet_service.cc:1467] Processing CreateTablet for tablet 6fde852ed1264555af856522b4abc093 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240426 09:54:42.669600 9929 tablet_service.cc:1467] Processing CreateTablet for tablet 7a3634dc56b34299b112ff8555d47c1e (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240426 09:54:42.711413 10137 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:42.711717 9929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a3634dc56b34299b112ff8555d47c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.712378 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:42.689013 10131 consensus_queue.cc:260] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:42.713629 9979 leader_election.cc:341] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:42.714540 10131 ts_tablet_manager.cc:1398] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.100s user 0.001s sys 0.006s
I20240426 09:54:42.714859 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:42.715389 10131 tablet_bootstrap.cc:492] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
W20240426 09:54:42.718199 9982 leader_election.cc:341] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:42.718595 9982 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:42.724431 10137 raft_consensus.cc:2726] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:42.726481 10130 ts_tablet_manager.cc:1398] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20240426 09:54:42.726444 9923 tablet_service.cc:1467] Processing CreateTablet for tablet f46890d4f3474e88835f7b3e08a30e1a (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240426 09:54:42.727835 9923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f46890d4f3474e88835f7b3e08a30e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.727936 10004 tablet_service.cc:1467] Processing CreateTablet for tablet 7a3634dc56b34299b112ff8555d47c1e (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240426 09:54:42.729074 10004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a3634dc56b34299b112ff8555d47c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.730854 10080 tablet_service.cc:1467] Processing CreateTablet for tablet ab7abcb2e80641eca676add7616cd966 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240426 09:54:42.731961 10080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7abcb2e80641eca676add7616cd966. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.735038 10131 tablet_bootstrap.cc:654] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:42.737607 10001 tablet_service.cc:1467] Processing CreateTablet for tablet 67dfdb63e28a460486dfe163f03bf820 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240426 09:54:42.747396 10001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67dfdb63e28a460486dfe163f03bf820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.751119 9922 tablet_service.cc:1467] Processing CreateTablet for tablet 8574d769068f477e859574c959d5880e (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240426 09:54:42.743975 10075 tablet_service.cc:1467] Processing CreateTablet for tablet 23f4904f98c241de901e223b5ab4725d (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240426 09:54:42.749262 10130 tablet_bootstrap.cc:492] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:42.750878 10132 tablet_bootstrap.cc:492] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:42.752285 10132 ts_tablet_manager.cc:1367] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.301s user 0.047s sys 0.019s
I20240426 09:54:42.755497 9926 tablet_service.cc:1467] Processing CreateTablet for tablet 5e11de7e65e544ffac5f7683e9614536 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240426 09:54:42.754927 10132 raft_consensus.cc:348] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.757755 10005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fde852ed1264555af856522b4abc093. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.758641 10075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f4904f98c241de901e223b5ab4725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.762185 9922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8574d769068f477e859574c959d5880e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.759883 10071 tablet_service.cc:1467] Processing CreateTablet for tablet fa1ede5c9b614a0f8b7adb20a7cfbbb1 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240426 09:54:42.762835 10076 tablet_service.cc:1467] Processing CreateTablet for tablet 60bebc132e4141bdbab199326986516e (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240426 09:54:42.763545 10071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1ede5c9b614a0f8b7adb20a7cfbbb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.768129 10130 tablet_bootstrap.cc:654] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:42.770854 10132 raft_consensus.cc:374] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:42.771214 10132 raft_consensus.cc:729] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:42.770849 9926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e11de7e65e544ffac5f7683e9614536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.774828 10132 consensus_queue.cc:260] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.787027 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:54:42.788221 9904 leader_election.cc:341] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:42.788877 10132 ts_tablet_manager.cc:1398] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20240426 09:54:42.789711 10132 tablet_bootstrap.cc:492] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:42.790185 10003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb385fda44e94153ae3cd9ca21240939. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.734627 10073 tablet_service.cc:1467] Processing CreateTablet for tablet a7ef86a50a514e8ba09c76ca674b2100 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240426 09:54:42.795835 10076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60bebc132e4141bdbab199326986516e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.798565 10132 tablet_bootstrap.cc:654] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:42.804514 9999 tablet_service.cc:1467] Processing CreateTablet for tablet f46890d4f3474e88835f7b3e08a30e1a (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240426 09:54:42.805660 9999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f46890d4f3474e88835f7b3e08a30e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.806454 10004 tablet_service.cc:1467] Processing CreateTablet for tablet 8574d769068f477e859574c959d5880e (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240426 09:54:42.807572 10004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8574d769068f477e859574c959d5880e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.812119 10075 tablet_service.cc:1467] Processing CreateTablet for tablet 7a3634dc56b34299b112ff8555d47c1e (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240426 09:54:42.819195 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ef86a50a514e8ba09c76ca674b2100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.823293 10006 tablet_service.cc:1467] Processing CreateTablet for tablet 5e11de7e65e544ffac5f7683e9614536 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240426 09:54:42.819178 10001 tablet_service.cc:1467] Processing CreateTablet for tablet 253cc496ee9644ffba6201074808dbf0 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240426 09:54:42.824462 10006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e11de7e65e544ffac5f7683e9614536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.825078 10001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253cc496ee9644ffba6201074808dbf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.827880 10080 tablet_service.cc:1467] Processing CreateTablet for tablet 67dfdb63e28a460486dfe163f03bf820 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240426 09:54:42.829038 10080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67dfdb63e28a460486dfe163f03bf820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.831583 10075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a3634dc56b34299b112ff8555d47c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.782361 10074 tablet_service.cc:1467] Processing CreateTablet for tablet fb385fda44e94153ae3cd9ca21240939 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240426 09:54:42.837121 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:42.843746 10078 tablet_service.cc:1467] Processing CreateTablet for tablet 951e4c48fcb642e58ab79652d3f86261 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240426 09:54:42.760289 9925 tablet_service.cc:1467] Processing CreateTablet for tablet 253cc496ee9644ffba6201074808dbf0 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240426 09:54:42.855332 10078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 951e4c48fcb642e58ab79652d3f86261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.857460 10079 tablet_service.cc:1467] Processing CreateTablet for tablet 6fde852ed1264555af856522b4abc093 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240426 09:54:42.859573 10079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fde852ed1264555af856522b4abc093. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.859827 10131 tablet_bootstrap.cc:492] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:42.860266 10131 ts_tablet_manager.cc:1367] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.145s user 0.008s sys 0.001s
I20240426 09:54:42.862262 10077 tablet_service.cc:1467] Processing CreateTablet for tablet f46890d4f3474e88835f7b3e08a30e1a (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240426 09:54:42.863008 9925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253cc496ee9644ffba6201074808dbf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.862380 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 8574d769068f477e859574c959d5880e (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
W20240426 09:54:42.862478 9906 leader_election.cc:341] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:42.872676 9906 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:42.873471 10139 raft_consensus.cc:2726] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:42.874610 10130 tablet_bootstrap.cc:492] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:42.877218 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 253cc496ee9644ffba6201074808dbf0 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240426 09:54:42.857685 10071 tablet_service.cc:1467] Processing CreateTablet for tablet 5e11de7e65e544ffac5f7683e9614536 (DEFAULT_TABLE table=test-workload [id=423e8fc5d5d743c08b5472616da5df50]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240426 09:54:42.878902 10130 ts_tablet_manager.cc:1367] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.130s user 0.008s sys 0.007s
I20240426 09:54:42.883359 10130 raft_consensus.cc:348] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.887089 10130 raft_consensus.cc:374] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:42.887765 10130 raft_consensus.cc:729] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:42.892040 10077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f46890d4f3474e88835f7b3e08a30e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.892724 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8574d769068f477e859574c959d5880e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.893941 10131 raft_consensus.cc:348] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.894657 10131 raft_consensus.cc:374] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:42.900458 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb385fda44e94153ae3cd9ca21240939. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.907508 10071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e11de7e65e544ffac5f7683e9614536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.911053 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253cc496ee9644ffba6201074808dbf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:42.912699 10132 tablet_bootstrap.cc:492] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:42.897657 10130 consensus_queue.cc:260] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.900733 10131 raft_consensus.cc:729] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:42.919176 10132 ts_tablet_manager.cc:1367] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.130s user 0.014s sys 0.007s
I20240426 09:54:42.919694 10131 consensus_queue.cc:260] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.921870 10132 raft_consensus.cc:348] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.923909 10130 ts_tablet_manager.cc:1398] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.043s user 0.006s sys 0.000s
I20240426 09:54:42.930399 10130 tablet_bootstrap.cc:492] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:42.924535 10131 ts_tablet_manager.cc:1398] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.064s user 0.006s sys 0.001s
I20240426 09:54:42.930944 10132 raft_consensus.cc:374] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:42.931325 10132 raft_consensus.cc:729] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:42.932056 10132 consensus_queue.cc:260] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.942322 10132 ts_tablet_manager.cc:1398] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.023s user 0.004s sys 0.002s
I20240426 09:54:42.943327 10132 tablet_bootstrap.cc:492] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:42.933367 10131 tablet_bootstrap.cc:492] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:42.935703 10130 tablet_bootstrap.cc:654] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:42.954818 10132 tablet_bootstrap.cc:654] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:42.958592 10131 tablet_bootstrap.cc:654] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:42.960005 10130 tablet_bootstrap.cc:492] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:42.962705 10136 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:42.963220 10136 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.964188 10130 ts_tablet_manager.cc:1367] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.034s user 0.017s sys 0.003s
I20240426 09:54:42.965087 10136 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:42.966471 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:42.966471 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:42.967487 9941 raft_consensus.cc:2454] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
W20240426 09:54:42.967787 10055 leader_election.cc:341] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:42.968616 10056 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:54:42.969188 10136 raft_consensus.cc:2781] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:42.969169 10130 raft_consensus.cc:348] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.969786 10136 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:42.966970 10132 tablet_bootstrap.cc:492] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:42.970206 10136 raft_consensus.cc:3037] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:42.970836 10132 ts_tablet_manager.cc:1367] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.000s
I20240426 09:54:42.971324 10130 raft_consensus.cc:374] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:42.971642 10130 raft_consensus.cc:729] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:42.972996 10132 raft_consensus.cc:348] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.973306 10131 tablet_bootstrap.cc:492] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:42.973778 10132 raft_consensus.cc:374] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:42.973912 10131 ts_tablet_manager.cc:1367] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.012s
I20240426 09:54:42.974246 10132 raft_consensus.cc:729] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:42.973841 10130 consensus_queue.cc:260] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.975193 10132 consensus_queue.cc:260] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.976192 10130 ts_tablet_manager.cc:1398] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20240426 09:54:42.977442 10130 tablet_bootstrap.cc:492] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:42.977460 10132 ts_tablet_manager.cc:1398] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 09:54:42.978675 10132 tablet_bootstrap.cc:492] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:42.983489 10130 tablet_bootstrap.cc:654] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:42.984035 10132 tablet_bootstrap.cc:654] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:42.985915 10131 raft_consensus.cc:348] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.986646 10131 raft_consensus.cc:374] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:42.986951 10131 raft_consensus.cc:729] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:42.987699 10131 consensus_queue.cc:260] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.995296 10137 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:42.995776 10137 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:42.997649 10137 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:42.997627 10131 ts_tablet_manager.cc:1398] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.023s user 0.010s sys 0.001s
I20240426 09:54:42.998669 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:54:42.999905 9982 leader_election.cc:341] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.001343 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:43.001699 10131 tablet_bootstrap.cc:492] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
W20240426 09:54:43.002581 9979 leader_election.cc:341] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.003021 9979 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:43.004041 10137 raft_consensus.cc:2726] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:43.006911 10131 tablet_bootstrap.cc:654] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.009016 10136 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.011445 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:43.011988 9941 raft_consensus.cc:3037] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.018841 9941 raft_consensus.cc:2454] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:43.019909 10056 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:43.020781 10162 raft_consensus.cc:2781] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:43.022187 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:54:43.023547 10055 leader_election.cc:341] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.024106 10136 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:43.027815 10132 tablet_bootstrap.cc:492] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:43.028261 10132 ts_tablet_manager.cc:1367] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.050s user 0.024s sys 0.021s
I20240426 09:54:43.028864 10130 tablet_bootstrap.cc:492] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:43.029301 10130 ts_tablet_manager.cc:1367] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.052s user 0.026s sys 0.023s
I20240426 09:54:43.030789 10132 raft_consensus.cc:348] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.031512 10132 raft_consensus.cc:374] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.031808 10132 raft_consensus.cc:729] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.031769 10130 raft_consensus.cc:348] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.032660 10130 raft_consensus.cc:374] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.033059 10130 raft_consensus.cc:729] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.032671 10132 consensus_queue.cc:260] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.033646 10131 tablet_bootstrap.cc:492] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:43.033946 10162 raft_consensus.cc:686] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:43.033805 10130 consensus_queue.cc:260] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.034976 10132 ts_tablet_manager.cc:1398] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240426 09:54:43.035881 10132 tablet_bootstrap.cc:492] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:43.036209 10130 ts_tablet_manager.cc:1398] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240426 09:54:43.037179 10130 tablet_bootstrap.cc:492] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:43.034723 10162 consensus_queue.cc:237] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.040534 10131 ts_tablet_manager.cc:1367] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.015s
I20240426 09:54:43.041317 10132 tablet_bootstrap.cc:654] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.042852 10130 tablet_bootstrap.cc:654] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.043148 10131 raft_consensus.cc:348] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.043848 10131 raft_consensus.cc:374] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.044103 10131 raft_consensus.cc:729] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.051303 10131 consensus_queue.cc:260] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.052601 10132 tablet_bootstrap.cc:492] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:43.053056 10132 ts_tablet_manager.cc:1367] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20240426 09:54:43.053457 10131 ts_tablet_manager.cc:1398] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.013s user 0.000s sys 0.004s
I20240426 09:54:43.053890 10130 tablet_bootstrap.cc:492] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:43.054337 10131 tablet_bootstrap.cc:492] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:43.054338 10130 ts_tablet_manager.cc:1367] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20240426 09:54:43.055276 10132 raft_consensus.cc:348] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.055789 10132 raft_consensus.cc:374] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.056005 10132 raft_consensus.cc:729] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.056528 10132 consensus_queue.cc:260] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.057173 10130 raft_consensus.cc:348] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.057929 10130 raft_consensus.cc:374] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.058212 10130 raft_consensus.cc:729] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.058473 10132 ts_tablet_manager.cc:1398] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240426 09:54:43.058962 10130 consensus_queue.cc:260] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.061168 10130 ts_tablet_manager.cc:1398] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240426 09:54:43.062242 10130 tablet_bootstrap.cc:492] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:43.064472 10131 tablet_bootstrap.cc:654] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.066082 9837 catalog_manager.cc:5484] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:43.068003 10130 tablet_bootstrap.cc:654] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.073921 10132 tablet_bootstrap.cc:492] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:43.083343 10132 tablet_bootstrap.cc:654] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
W20240426 09:54:43.083534 9887 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 09:54:43.093057 10132 tablet_bootstrap.cc:492] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:43.093575 10132 ts_tablet_manager.cc:1367] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.001s
I20240426 09:54:43.093794 10131 tablet_bootstrap.cc:492] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:43.094311 10131 ts_tablet_manager.cc:1367] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.040s user 0.015s sys 0.019s
I20240426 09:54:43.093832 10130 tablet_bootstrap.cc:492] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:43.095678 10132 raft_consensus.cc:348] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.096299 10132 raft_consensus.cc:374] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.096379 10130 ts_tablet_manager.cc:1367] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.015s
I20240426 09:54:43.096619 10132 raft_consensus.cc:729] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.096936 10131 raft_consensus.cc:348] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.097273 10132 consensus_queue.cc:260] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.097684 10131 raft_consensus.cc:374] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.098096 10131 raft_consensus.cc:729] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.099200 10132 ts_tablet_manager.cc:1398] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20240426 09:54:43.098888 10131 consensus_queue.cc:260] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.099161 10130 raft_consensus.cc:348] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.099975 10130 raft_consensus.cc:374] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.100271 10130 raft_consensus.cc:729] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.100364 10132 tablet_bootstrap.cc:492] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:43.101298 10131 ts_tablet_manager.cc:1398] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240426 09:54:43.101047 10130 consensus_queue.cc:260] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.102414 10131 tablet_bootstrap.cc:492] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:43.108004 10131 tablet_bootstrap.cc:654] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.108433 10130 ts_tablet_manager.cc:1398] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.012s user 0.001s sys 0.004s
I20240426 09:54:43.109347 10130 tablet_bootstrap.cc:492] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:43.110675 10132 tablet_bootstrap.cc:654] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.115252 10130 tablet_bootstrap.cc:654] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.119132 10131 tablet_bootstrap.cc:492] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:43.119450 10139 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.119613 10131 ts_tablet_manager.cc:1367] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.007s
I20240426 09:54:43.119930 10139 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.121850 10139 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.122427 10131 raft_consensus.cc:348] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.123502 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:43.123925 10131 raft_consensus.cc:374] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.124285 10131 raft_consensus.cc:729] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
W20240426 09:54:43.124776 9904 leader_election.cc:341] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.125000 10131 consensus_queue.cc:260] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.123982 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:43.127576 10131 ts_tablet_manager.cc:1398] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240426 09:54:43.127964 10016 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:43.128504 10131 tablet_bootstrap.cc:492] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:43.128998 10130 tablet_bootstrap.cc:492] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:43.129424 10130 ts_tablet_manager.cc:1367] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.000s
I20240426 09:54:43.129263 9906 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:43.130417 10139 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:43.130831 10139 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:43.131210 10139 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.132010 10130 raft_consensus.cc:348] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.132664 10130 raft_consensus.cc:374] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.132927 10130 raft_consensus.cc:729] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.133654 10130 consensus_queue.cc:260] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.134285 10131 tablet_bootstrap.cc:654] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.137670 10132 tablet_bootstrap.cc:492] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:43.138087 10132 ts_tablet_manager.cc:1367] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.000s
I20240426 09:54:43.140578 10132 raft_consensus.cc:348] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.141206 10132 raft_consensus.cc:374] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.141492 10132 raft_consensus.cc:729] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.141805 10139 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.142132 10132 consensus_queue.cc:260] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.143115 10130 ts_tablet_manager.cc:1398] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.013s user 0.007s sys 0.007s
I20240426 09:54:43.145085 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:43.146008 10131 tablet_bootstrap.cc:492] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:43.146273 10130 tablet_bootstrap.cc:492] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:43.146435 10131 ts_tablet_manager.cc:1367] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.002s
I20240426 09:54:43.146456 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:43.147135 10016 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.148828 10131 raft_consensus.cc:348] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.149477 10131 raft_consensus.cc:374] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.149713 10131 raft_consensus.cc:729] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.150364 10131 consensus_queue.cc:260] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.151674 10130 tablet_bootstrap.cc:654] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
W20240426 09:54:43.153065 9904 leader_election.cc:341] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.156386 10016 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:43.157343 9906 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:43.157908 10139 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.158210 10178 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:43.158658 10178 raft_consensus.cc:686] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:54:43.159147 10131 ts_tablet_manager.cc:1398] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240426 09:54:43.159484 10132 ts_tablet_manager.cc:1398] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20240426 09:54:43.159999 10131 tablet_bootstrap.cc:492] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:43.159468 10178 consensus_queue.cc:237] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.160440 10132 tablet_bootstrap.cc:492] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:43.164927 10131 tablet_bootstrap.cc:654] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.165468 10132 tablet_bootstrap.cc:654] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.168314 10137 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.168788 10137 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.170701 10137 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.171661 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:43.172173 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.172821 9941 raft_consensus.cc:2454] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
W20240426 09:54:43.172940 9979 leader_election.cc:341] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.174099 9982 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:43.175172 10137 raft_consensus.cc:2781] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:43.175487 10137 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:43.175793 10137 raft_consensus.cc:3037] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.178298 10131 tablet_bootstrap.cc:492] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:43.178709 10131 ts_tablet_manager.cc:1367] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.000s
I20240426 09:54:43.179044 10132 tablet_bootstrap.cc:492] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:43.180080 10132 ts_tablet_manager.cc:1367] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.007s
I20240426 09:54:43.180964 10131 raft_consensus.cc:348] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.181631 10131 raft_consensus.cc:374] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.181869 10131 raft_consensus.cc:729] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.182626 10131 consensus_queue.cc:260] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.184285 10130 tablet_bootstrap.cc:492] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:43.184693 10130 ts_tablet_manager.cc:1367] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.007s
I20240426 09:54:43.184787 10131 ts_tablet_manager.cc:1398] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240426 09:54:43.184298 9839 catalog_manager.cc:5484] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:43.185628 10131 tablet_bootstrap.cc:492] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:43.185096 10132 raft_consensus.cc:348] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.186439 10132 raft_consensus.cc:374] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.186699 10132 raft_consensus.cc:729] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.188129 10132 consensus_queue.cc:260] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.190557 10131 tablet_bootstrap.cc:654] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.186978 10130 raft_consensus.cc:348] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.190965 10130 raft_consensus.cc:374] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.191238 10130 raft_consensus.cc:729] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.191982 10130 consensus_queue.cc:260] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.194139 10130 ts_tablet_manager.cc:1398] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240426 09:54:43.194743 10184 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.197260 10184 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.199261 10184 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.195052 10130 tablet_bootstrap.cc:492] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:43.200266 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:54:43.201364 9979 leader_election.cc:341] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.201952 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.196713 10132 ts_tablet_manager.cc:1398] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
W20240426 09:54:43.203266 9982 leader_election.cc:341] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.203662 9982 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:43.196769 10178 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.204317 10178 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.205766 10130 tablet_bootstrap.cc:654] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.206291 10178 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.203611 10132 tablet_bootstrap.cc:492] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:43.208130 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:54:43.209364 9904 leader_election.cc:341] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.213202 10132 tablet_bootstrap.cc:654] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.213538 10184 raft_consensus.cc:2726] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:43.213379 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:43.213083 10137 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.214432 10016 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:43.216082 9906 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:43.216663 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:43.217144 10178 raft_consensus.cc:2781] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:43.217631 10178 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:43.217984 10178 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.218441 10137 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.217221 9941 raft_consensus.cc:3037] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.218978 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
W20240426 09:54:43.220167 9979 leader_election.cc:341] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.225721 9941 raft_consensus.cc:2454] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:43.226994 9982 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:43.228128 10137 raft_consensus.cc:2781] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:43.228576 10137 raft_consensus.cc:686] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:43.229362 10137 consensus_queue.cc:237] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.225519 10178 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.235522 10132 tablet_bootstrap.cc:492] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:43.235951 10132 ts_tablet_manager.cc:1367] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.003s
I20240426 09:54:43.236563 10131 tablet_bootstrap.cc:492] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:43.236963 10131 ts_tablet_manager.cc:1367] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.051s user 0.028s sys 0.015s
I20240426 09:54:43.236975 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:43.237608 10016 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.238178 10132 raft_consensus.cc:348] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.238832 10132 raft_consensus.cc:374] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.239073 10132 raft_consensus.cc:729] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.239702 10132 consensus_queue.cc:260] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.240159 10131 raft_consensus.cc:348] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.240813 10131 raft_consensus.cc:374] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.241089 10131 raft_consensus.cc:729] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.241601 10132 ts_tablet_manager.cc:1398] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240426 09:54:43.241771 10131 consensus_queue.cc:260] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.243062 10130 tablet_bootstrap.cc:492] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:43.243466 10130 ts_tablet_manager.cc:1367] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.019s
I20240426 09:54:43.244730 10131 ts_tablet_manager.cc:1398] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240426 09:54:43.245561 10131 tablet_bootstrap.cc:492] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:43.246284 10132 tablet_bootstrap.cc:492] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:43.256215 10016 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:43.256721 10131 tablet_bootstrap.cc:654] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.257282 9906 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:43.257688 10132 tablet_bootstrap.cc:654] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.258795 10178 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.245630 10130 raft_consensus.cc:348] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.259305 10139 raft_consensus.cc:2781] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:43.259579 10130 raft_consensus.cc:374] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.259712 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:43.259793 10139 raft_consensus.cc:686] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
W20240426 09:54:43.261022 9904 leader_election.cc:341] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.259891 10130 raft_consensus.cc:729] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.261292 10139 consensus_queue.cc:237] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.262055 10130 consensus_queue.cc:260] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.266821 10132 tablet_bootstrap.cc:492] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:43.267242 10132 ts_tablet_manager.cc:1367] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20240426 09:54:43.269606 10132 raft_consensus.cc:348] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.270238 10132 raft_consensus.cc:374] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.270501 10132 raft_consensus.cc:729] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.271143 10131 tablet_bootstrap.cc:492] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:43.271723 10131 ts_tablet_manager.cc:1367] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.013s
I20240426 09:54:43.270946 9839 catalog_manager.cc:5484] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:43.271184 10132 consensus_queue.cc:260] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.272014 10130 ts_tablet_manager.cc:1398] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.028s user 0.009s sys 0.001s
I20240426 09:54:43.272866 10130 tablet_bootstrap.cc:492] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:43.273631 10132 ts_tablet_manager.cc:1398] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240426 09:54:43.274523 10132 tablet_bootstrap.cc:492] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:43.274331 10131 raft_consensus.cc:348] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.274995 10131 raft_consensus.cc:374] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.275262 10131 raft_consensus.cc:729] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.275905 10131 consensus_queue.cc:260] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.278105 10130 tablet_bootstrap.cc:654] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.280139 10132 tablet_bootstrap.cc:654] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.285023 10131 ts_tablet_manager.cc:1398] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.013s user 0.002s sys 0.003s
I20240426 09:54:43.289024 10130 tablet_bootstrap.cc:492] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:43.289150 10132 tablet_bootstrap.cc:492] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:43.289501 10130 ts_tablet_manager.cc:1367] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.004s
I20240426 09:54:43.289583 10132 ts_tablet_manager.cc:1367] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20240426 09:54:43.291873 10132 raft_consensus.cc:348] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.292063 10130 raft_consensus.cc:348] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.292634 10132 raft_consensus.cc:374] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.292768 10130 raft_consensus.cc:374] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.292959 10131 tablet_bootstrap.cc:492] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:43.293116 10130 raft_consensus.cc:729] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.293032 9837 catalog_manager.cc:5484] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:43.293913 10130 consensus_queue.cc:260] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.295894 10130 ts_tablet_manager.cc:1398] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 09:54:43.296320 10139 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.296697 10130 tablet_bootstrap.cc:492] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:43.296754 10139 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.298718 10131 tablet_bootstrap.cc:654] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.292973 10132 raft_consensus.cc:729] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.299579 10132 consensus_queue.cc:260] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.301479 10132 ts_tablet_manager.cc:1398] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20240426 09:54:43.301702 10130 tablet_bootstrap.cc:654] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.302492 10132 tablet_bootstrap.cc:492] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:43.304528 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:43.305380 10090 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:43.306577 9904 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:43.307490 10193 raft_consensus.cc:2781] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:43.307848 10193 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:43.308156 10193 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.308420 10132 tablet_bootstrap.cc:654] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.298692 10139 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.301441 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:43.311230 10016 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:43.313432 10139 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.313882 10139 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.314859 10193 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.316455 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:43.316882 10132 tablet_bootstrap.cc:492] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:43.317286 10132 ts_tablet_manager.cc:1367] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20240426 09:54:43.317920 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:43.318518 10015 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.318609 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:43.319725 10132 raft_consensus.cc:348] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:43.320101 9904 leader_election.cc:341] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.320384 10132 raft_consensus.cc:374] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.320667 10132 raft_consensus.cc:729] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.320758 10139 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.321259 10132 consensus_queue.cc:260] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:43.323242 9906 leader_election.cc:341] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.323494 10132 ts_tablet_manager.cc:1398] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240426 09:54:43.324348 10132 tablet_bootstrap.cc:492] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:43.323661 9906 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:43.327709 10131 tablet_bootstrap.cc:492] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:43.328033 10131 ts_tablet_manager.cc:1367] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.013s
I20240426 09:54:43.328585 10015 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:43.329646 10131 raft_consensus.cc:348] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.329856 10132 tablet_bootstrap.cc:654] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.330127 10131 raft_consensus.cc:374] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.330384 10131 raft_consensus.cc:729] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.330897 10131 consensus_queue.cc:260] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.333192 10139 raft_consensus.cc:2726] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:43.334355 9906 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:43.334647 10132 tablet_bootstrap.cc:492] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:43.335175 10132 ts_tablet_manager.cc:1367] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240426 09:54:43.335148 10139 raft_consensus.cc:2781] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:43.336517 10139 raft_consensus.cc:686] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:54:43.337808 10132 raft_consensus.cc:348] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.338404 10178 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.338618 10132 raft_consensus.cc:374] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.339059 10132 raft_consensus.cc:729] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.338907 10178 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.340126 10162 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.339946 10132 consensus_queue.cc:260] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.340783 10178 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.340530 10162 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.342077 10132 ts_tablet_manager.cc:1398] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 09:54:43.342952 10132 tablet_bootstrap.cc:492] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:43.342641 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:43.343442 10130 tablet_bootstrap.cc:492] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:43.342976 10162 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:43.343863 10130 ts_tablet_manager.cc:1367] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.047s user 0.009s sys 0.003s
I20240426 09:54:43.339717 10131 ts_tablet_manager.cc:1398] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240426 09:54:43.344396 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.343612 10090 raft_consensus.cc:2454] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:43.346060 9904 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:43.346201 10130 raft_consensus.cc:348] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.346874 10130 raft_consensus.cc:374] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.347126 10178 raft_consensus.cc:2781] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:43.347151 10130 raft_consensus.cc:729] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.347613 10178 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:43.347955 10178 raft_consensus.cc:3037] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.348523 10131 tablet_bootstrap.cc:492] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:43.349207 10132 tablet_bootstrap.cc:654] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
W20240426 09:54:43.351264 10056 leader_election.cc:341] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.350977 10130 consensus_queue.cc:260] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.352593 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:43.353153 10016 raft_consensus.cc:2454] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:43.354853 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:54:43.355942 10055 leader_election.cc:341] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.356314 10055 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:43.357267 10162 raft_consensus.cc:2726] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:43.356958 10178 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.337700 10139 consensus_queue.cc:237] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.360059 10193 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.360575 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:43.360976 10090 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.363651 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:43.364207 10015 raft_consensus.cc:3037] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.365773 10130 ts_tablet_manager.cc:1398] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.022s user 0.006s sys 0.007s
I20240426 09:54:43.366672 10130 tablet_bootstrap.cc:492] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:43.366878 10162 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.368352 10162 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.370219 10162 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:43.370920 10178 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.371407 10090 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:43.371908 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:43.373845 10137 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.374567 10137 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.372712 10016 raft_consensus.cc:2379] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 1: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:54:43.376482 10137 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.376677 10162 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.377285 10130 tablet_bootstrap.cc:654] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.377146 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.379771 10015 raft_consensus.cc:2454] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:43.379676 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:43.380182 10056 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:43.380473 10089 raft_consensus.cc:2454] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:43.381359 9906 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:43.382473 10136 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.382743 10164 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.382985 10178 raft_consensus.cc:2781] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:43.382951 10136 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.383697 10178 raft_consensus.cc:686] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:54:43.384033 9979 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:43.384778 10137 raft_consensus.cc:2781] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:43.384833 10136 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:43.385210 10137 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:43.377141 10162 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.385555 10137 raft_consensus.cc:3037] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.386658 10162 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:43.388049 10131 tablet_bootstrap.cc:654] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.382486 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.389515 9941 raft_consensus.cc:2454] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:43.389833 10132 tablet_bootstrap.cc:492] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:43.390234 10132 ts_tablet_manager.cc:1367] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.022s
I20240426 09:54:43.390360 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:43.391388 10137 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.392706 10132 raft_consensus.cc:348] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.393239 10137 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.393393 10132 raft_consensus.cc:374] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.393702 10132 raft_consensus.cc:729] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.393996 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:43.393254 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.395206 9940 raft_consensus.cc:2379] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 1: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:54:43.394471 9837 catalog_manager.cc:5484] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
W20240426 09:54:43.395198 10055 leader_election.cc:341] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.396049 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:43.394332 10132 consensus_queue.cc:260] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.396538 10089 raft_consensus.cc:3037] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.397727 10132 ts_tablet_manager.cc:1398] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240426 09:54:43.398799 10132 tablet_bootstrap.cc:492] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:43.399067 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.400707 10162 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.401139 10162 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.403324 10056 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:43.403939 10132 tablet_bootstrap.cc:654] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.404028 10136 raft_consensus.cc:3037] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
W20240426 09:54:43.406333 10056 leader_election.cc:341] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.407047 10056 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:43.408798 10131 tablet_bootstrap.cc:492] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:43.409271 10131 ts_tablet_manager.cc:1367] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.061s user 0.013s sys 0.003s
I20240426 09:54:43.411567 10131 raft_consensus.cc:348] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.411870 10137 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.412544 10184 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.412448 10137 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.412961 10184 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.414496 10162 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:43.414882 10137 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.415002 10162 raft_consensus.cc:2726] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:43.415158 10184 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.416069 10131 raft_consensus.cc:374] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.416432 10131 raft_consensus.cc:729] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.417126 10131 consensus_queue.cc:260] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.419413 10164 raft_consensus.cc:2726] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:43.420701 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:43.421268 10015 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:43.421253 10132 tablet_bootstrap.cc:492] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:43.422298 10132 ts_tablet_manager.cc:1367] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.011s
I20240426 09:54:43.423012 10131 ts_tablet_manager.cc:1398] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240426 09:54:43.423920 10131 tablet_bootstrap.cc:492] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:43.424630 10132 raft_consensus.cc:348] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.425328 10132 raft_consensus.cc:374] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.425621 10132 raft_consensus.cc:729] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.426223 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:43.426342 10132 consensus_queue.cc:260] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.429260 10131 tablet_bootstrap.cc:654] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.432077 10132 ts_tablet_manager.cc:1398] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240426 09:54:43.432919 10132 tablet_bootstrap.cc:492] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:43.435979 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:43.436518 9941 raft_consensus.cc:3037] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.437803 10132 tablet_bootstrap.cc:654] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.384457 10178 consensus_queue.cc:237] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.440521 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.441581 9940 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:43.443130 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:43.445205 9941 raft_consensus.cc:2454] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:43.446851 10136 raft_consensus.cc:2726] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:43.448271 10055 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
W20240426 09:54:43.449450 9979 leader_election.cc:341] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.449797 10136 raft_consensus.cc:2781] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:43.450119 10136 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:43.450429 10136 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.451081 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.451741 9940 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:43.451109 9982 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:43.453408 10184 raft_consensus.cc:2781] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:43.453749 10184 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:43.454064 10184 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.453848 9982 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:43.455305 10137 raft_consensus.cc:2781] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:43.456053 10137 raft_consensus.cc:686] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:43.457270 10137 consensus_queue.cc:237] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.458071 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.458734 10131 tablet_bootstrap.cc:492] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:43.459211 10131 ts_tablet_manager.cc:1367] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.006s
I20240426 09:54:43.382488 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:43.462029 10136 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.462828 10131 raft_consensus.cc:348] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.463644 10131 raft_consensus.cc:374] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.463901 10131 raft_consensus.cc:729] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.464592 10131 consensus_queue.cc:260] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.472445 10090 raft_consensus.cc:3037] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.475723 9941 raft_consensus.cc:2379] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:54:43.485733 10131 ts_tablet_manager.cc:1398] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.026s user 0.004s sys 0.003s
I20240426 09:54:43.475003 10136 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:43.486838 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:43.487403 10015 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.493158 10184 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.495833 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:43.495307 9839 catalog_manager.cc:5484] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:43.496400 9940 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.497501 10131 tablet_bootstrap.cc:492] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:43.505861 10184 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.505980 9982 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:43.507452 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
W20240426 09:54:43.508710 9979 leader_election.cc:341] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.509166 10089 raft_consensus.cc:2454] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:43.514575 10130 tablet_bootstrap.cc:492] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:43.514940 10130 ts_tablet_manager.cc:1367] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.148s user 0.035s sys 0.046s
I20240426 09:54:43.516808 10130 raft_consensus.cc:348] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.517263 10130 raft_consensus.cc:374] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.517457 10130 raft_consensus.cc:729] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.517956 10130 consensus_queue.cc:260] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.518944 10184 raft_consensus.cc:3037] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.520166 10130 ts_tablet_manager.cc:1398] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20240426 09:54:43.520364 10136 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.521212 10130 tablet_bootstrap.cc:492] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:43.520861 10136 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.525964 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:54:43.486371 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 80065.8us
I20240426 09:54:43.532922 10184 raft_consensus.cc:2726] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:43.534128 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.534909 10162 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.535351 10162 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.537209 10162 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:43.537691 9939 raft_consensus.cc:2454] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:43.549707 10015 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:43.552505 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:43.553131 10014 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:43.554279 10055 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:43.556947 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.557608 10055 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:43.529517 10137 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.530441 10136 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:43.561347 10162 raft_consensus.cc:2781] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:43.590673 10162 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:43.591183 10162 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.553622 10056 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:43.596352 10130 tablet_bootstrap.cc:654] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.561424 10206 raft_consensus.cc:2781] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:43.592828 10016 raft_consensus.cc:2454] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:43.597167 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:43.567886 10137 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.599614 9941 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.599220 10206 raft_consensus.cc:686] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:43.600886 10131 tablet_bootstrap.cc:654] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.569603 9940 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
W20240426 09:54:43.598174 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 44635.2us
I20240426 09:54:43.589958 9939 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:43.603035 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.604298 10164 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.604728 10164 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:43.605140 9982 leader_election.cc:341] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.611531 10164 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:43.612051 10164 consensus_queue.cc:1035] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:43.612941 10178 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.602891 10136 raft_consensus.cc:2781] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:43.613358 10178 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.615343 10090 raft_consensus.cc:2454] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:43.620973 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:43.615346 10137 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.619191 9982 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:43.621920 10137 raft_consensus.cc:2781] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:43.617575 10136 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:43.619927 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:43.602461 10206 consensus_queue.cc:237] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.622355 10137 raft_consensus.cc:686] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:43.626554 10137 consensus_queue.cc:237] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.620040 10178 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.624563 10132 tablet_bootstrap.cc:492] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
W20240426 09:54:43.622306 9904 leader_election.cc:341] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.624671 10136 raft_consensus.cc:3037] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.625466 10184 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.628769 10184 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.621338 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.630578 10132 ts_tablet_manager.cc:1367] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.198s user 0.010s sys 0.009s
I20240426 09:54:43.625630 10016 raft_consensus.cc:2454] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:43.630537 10184 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.631206 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.631626 9906 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
W20240426 09:54:43.632387 9982 leader_election.cc:341] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.633055 10178 raft_consensus.cc:2781] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:43.633509 10178 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:43.631028 9839 catalog_manager.cc:5484] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:43.634313 10178 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.635995 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:43.636260 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:43.640201 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:54:43.640297 9979 leader_election.cc:341] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.640740 9979 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:43.641904 10132 raft_consensus.cc:348] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.642585 10132 raft_consensus.cc:374] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.642877 10132 raft_consensus.cc:729] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.643473 10132 consensus_queue.cc:260] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.636845 10089 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:43.641732 10184 raft_consensus.cc:2726] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:43.649585 10132 ts_tablet_manager.cc:1398] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20240426 09:54:43.650550 10132 tablet_bootstrap.cc:492] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:43.648878 10016 raft_consensus.cc:2379] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 1: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:54:43.654858 9979 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:43.657447 10184 raft_consensus.cc:2781] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:43.660945 10055 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:43.662029 10185 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.662453 10185 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.662420 10178 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.665880 10211 consensus_queue.cc:1035] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240426 09:54:43.667014 10185 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.667454 10185 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.667580 10211 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.667778 9941 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:43.663697 10184 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:43.668329 10184 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.669288 10130 tablet_bootstrap.cc:492] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:43.669692 10130 ts_tablet_manager.cc:1367] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.149s user 0.013s sys 0.035s
I20240426 09:54:43.673115 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.673509 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:43.673509 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:43.674844 10016 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.674716 10130 raft_consensus.cc:348] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.675397 10130 raft_consensus.cc:374] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.675650 10130 raft_consensus.cc:729] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.678006 10130 consensus_queue.cc:260] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.677353 9839 catalog_manager.cc:5484] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:43.713595 10162 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.715243 10130 ts_tablet_manager.cc:1398] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.045s user 0.027s sys 0.011s
I20240426 09:54:43.716117 10130 tablet_bootstrap.cc:492] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:43.716728 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:43.717283 10015 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.717605 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:43.718217 9941 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.718456 10132 tablet_bootstrap.cc:654] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.718711 10162 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:54:43.674322 9982 leader_election.cc:341] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.678720 10164 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.720490 10164 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.709573 10184 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.722323 10184 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.723162 10164 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:43.707984 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:43.677119 10178 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.727591 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:54:43.725020 9979 leader_election.cc:341] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
W20240426 09:54:43.729993 10056 leader_election.cc:341] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.725960 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:43.731982 10211 raft_consensus.cc:2726] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:43.732527 10131 tablet_bootstrap.cc:492] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:43.732892 10131 ts_tablet_manager.cc:1367] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.236s user 0.017s sys 0.009s
I20240426 09:54:43.732924 10016 raft_consensus.cc:2454] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:43.667837 10185 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.735412 10131 raft_consensus.cc:348] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.736279 10131 raft_consensus.cc:374] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.736574 10131 raft_consensus.cc:729] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
W20240426 09:54:43.731982 9904 leader_election.cc:341] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.737666 10131 consensus_queue.cc:260] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.739394 10131 ts_tablet_manager.cc:1398] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 09:54:43.739970 10131 tablet_bootstrap.cc:492] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:43.741456 10185 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.742632 10013 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:43.745028 9941 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:43.746304 10056 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:43.747076 10015 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:43.749814 10055 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:43.751045 10178 consensus_queue.cc:1035] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:43.751314 10211 raft_consensus.cc:2781] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:43.751659 10211 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:43.751951 10211 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.752724 10136 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.731530 9982 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:43.750723 10164 raft_consensus.cc:2781] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:43.754962 10164 raft_consensus.cc:686] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
W20240426 09:54:43.755172 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 76207us
I20240426 09:54:43.756206 9906 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:43.757333 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:43.757522 10185 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.757900 10090 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.757930 10185 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.759068 10178 raft_consensus.cc:2781] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:43.759521 10178 raft_consensus.cc:686] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:54:43.759807 10185 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.760061 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:43.760286 10178 consensus_queue.cc:237] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.762228 10185 raft_consensus.cc:2726] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240426 09:54:43.763170 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 88882.6us
I20240426 09:54:43.755843 10164 consensus_queue.cc:237] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.772686 10130 tablet_bootstrap.cc:654] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.773473 9837 catalog_manager.cc:5484] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:43.776865 10136 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:43.778151 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:43.778712 9941 raft_consensus.cc:3037] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.799772 9941 raft_consensus.cc:2454] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:43.805651 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:43.806353 10200 consensus_queue.cc:1035] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20240426 09:54:43.799901 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:43.831849 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:43.832557 10013 raft_consensus.cc:3037] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.834561 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.835218 10090 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:43.836439 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.838111 10217 consensus_queue.cc:1035] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240426 09:54:43.838505 9940 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:43.839547 10056 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:43.841243 10132 tablet_bootstrap.cc:492] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:43.841743 10132 ts_tablet_manager.cc:1367] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.191s user 0.016s sys 0.004s
I20240426 09:54:43.840914 10193 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240426 09:54:43.840360 10086 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:43.842334 10162 raft_consensus.cc:2781] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:43.854358 10162 raft_consensus.cc:686] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:43.856503 10131 tablet_bootstrap.cc:654] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.856619 10132 raft_consensus.cc:348] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.857251 10132 raft_consensus.cc:374] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.857484 10132 raft_consensus.cc:729] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.857985 10132 consensus_queue.cc:260] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.863209 10132 ts_tablet_manager.cc:1398] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20240426 09:54:43.864015 10132 tablet_bootstrap.cc:492] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:43.864091 9839 catalog_manager.cc:5484] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:43.869297 10178 consensus_queue.cc:1035] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 09:54:43.869364 10162 consensus_queue.cc:237] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.842702 10211 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.860942 9979 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:43.872059 10013 raft_consensus.cc:2454] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
W20240426 09:54:43.876425 9982 leader_election.cc:341] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.878254 10185 raft_consensus.cc:2781] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
W20240426 09:54:43.878921 9982 leader_election.cc:341] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.879734 9982 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:43.880631 10217 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:43.880991 10217 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:43.881289 10217 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.878713 10185 raft_consensus.cc:686] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:43.884135 10088 raft_consensus.cc:2379] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:54:43.884255 10185 consensus_queue.cc:237] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.884960 10211 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:43.893342 10217 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.889075 10178 consensus_queue.cc:1035] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240426 09:54:43.895207 10217 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.895747 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:43.896173 10016 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.895398 9838 catalog_manager.cc:5484] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
W20240426 09:54:43.889998 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 45341.4us
I20240426 09:54:43.902945 10130 tablet_bootstrap.cc:492] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:43.903307 10130 ts_tablet_manager.cc:1367] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.187s user 0.012s sys 0.009s
I20240426 09:54:43.905444 10200 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:43.905726 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:43.905733 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:43.907294 10086 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.915952 10130 raft_consensus.cc:348] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.916729 10130 raft_consensus.cc:374] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.916990 10130 raft_consensus.cc:729] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.920845 10130 consensus_queue.cc:260] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.931538 10217 consensus_queue.cc:1035] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:43.914961 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:43.935086 10132 tablet_bootstrap.cc:654] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:43.942534 10211 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.942638 10130 ts_tablet_manager.cc:1398] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
I20240426 09:54:43.920933 10016 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:43.943406 9941 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.952817 10131 tablet_bootstrap.cc:492] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:43.953299 10131 ts_tablet_manager.cc:1367] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.213s user 0.007s sys 0.008s
I20240426 09:54:43.955047 10217 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:43.955528 10217 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.957370 10217 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.959206 10086 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:43.960850 10130 tablet_bootstrap.cc:492] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:43.959687 10131 raft_consensus.cc:348] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.961629 10131 raft_consensus.cc:374] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:43.922268 10185 consensus_queue.cc:1035] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20240426 09:54:43.961932 10131 raft_consensus.cc:729] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:43.943063 10211 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.967926 9939 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:43.977568 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.978703 10193 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:54:43.978905 9982 leader_election.cc:341] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:43.979225 10193 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.971201 10055 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:43.981153 10136 raft_consensus.cc:2781] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:43.981187 10193 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:43.981706 10136 raft_consensus.cc:686] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:43.982496 10136 consensus_queue.cc:237] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:43.982463 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:43.988011 10086 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:43.989380 9904 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:43.992843 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:43.993227 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:43.993412 9938 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:43.993786 10013 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:43.994575 10056 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:43.994930 10211 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:43.995378 10211 raft_consensus.cc:2781] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:43.995694 10211 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:43.995836 10193 raft_consensus.cc:2781] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:43.996198 10193 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:43.996495 10193 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:44.000478 9941 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:44.001408 9982 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:44.002194 10217 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:44.002611 10217 raft_consensus.cc:686] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:44.003067 10131 consensus_queue.cc:260] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.003391 10217 consensus_queue.cc:237] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.005182 10131 ts_tablet_manager.cc:1398] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.052s user 0.002s sys 0.003s
I20240426 09:54:43.996026 10211 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:44.007495 10131 tablet_bootstrap.cc:492] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:43.971654 9979 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:44.019662 10185 raft_consensus.cc:2726] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:44.033170 10193 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.039085 10162 consensus_queue.cc:1035] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:44.045902 9939 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:44.057699 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:44.064903 10185 consensus_queue.cc:1035] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240426 09:54:44.070386 10131 tablet_bootstrap.cc:654] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.065940 10193 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:44.079713 10130 tablet_bootstrap.cc:654] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.042526 10211 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.085351 10090 raft_consensus.cc:2379] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
W20240426 09:54:44.093245 9887 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 09:54:44.093845 10211 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:44.094390 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:44.102051 10013 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:44.095006 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:44.087880 9839 catalog_manager.cc:5484] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:44.109864 9938 raft_consensus.cc:2379] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 1: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:54:44.101739 10136 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:44.112429 10136 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.115316 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:44.117538 10013 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:44.117720 10136 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:54:44.066202 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 101289us
I20240426 09:54:44.121078 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:44.125828 10055 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:44.126554 10211 raft_consensus.cc:2781] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:44.126928 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:44.127053 10211 raft_consensus.cc:686] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:44.132242 10211 consensus_queue.cc:237] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.153467 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:44.154883 10016 raft_consensus.cc:2379] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:54:44.156358 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:44.157112 10015 raft_consensus.cc:2379] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:54:44.180737 10200 consensus_queue.cc:1035] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:44.183682 10132 tablet_bootstrap.cc:492] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:44.184060 10132 ts_tablet_manager.cc:1367] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.320s user 0.030s sys 0.007s
W20240426 09:54:44.185185 9979 leader_election.cc:341] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:44.185602 9979 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:44.186414 10217 raft_consensus.cc:2726] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:44.187562 10185 consensus_queue.cc:1035] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:44.188956 10132 raft_consensus.cc:348] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.189630 10132 raft_consensus.cc:374] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.189877 10132 raft_consensus.cc:729] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.190428 10132 consensus_queue.cc:260] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.193549 9839 catalog_manager.cc:5484] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:44.197455 10056 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:44.198173 10206 raft_consensus.cc:2726] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:44.198339 10164 consensus_queue.cc:1035] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:44.200510 10132 ts_tablet_manager.cc:1398] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.016s user 0.004s sys 0.002s
I20240426 09:54:44.200650 10131 tablet_bootstrap.cc:492] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:44.201011 10131 ts_tablet_manager.cc:1367] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.194s user 0.031s sys 0.065s
I20240426 09:54:44.202919 10132 tablet_bootstrap.cc:492] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:44.213073 9906 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:44.214197 10200 raft_consensus.cc:2726] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240426 09:54:44.216468 10131 raft_consensus.cc:348] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.217332 10131 raft_consensus.cc:374] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.217589 10131 raft_consensus.cc:729] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.219976 10131 consensus_queue.cc:260] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.226511 10200 consensus_queue.cc:1035] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:44.227160 10131 ts_tablet_manager.cc:1398] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.026s user 0.003s sys 0.003s
I20240426 09:54:44.228070 10164 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:44.228569 10164 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.229239 10132 tablet_bootstrap.cc:654] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.230741 10164 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:44.231340 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:44.231729 10130 tablet_bootstrap.cc:492] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:44.232153 10130 ts_tablet_manager.cc:1367] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.271s user 0.013s sys 0.007s
I20240426 09:54:44.232437 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:54:44.232789 10056 leader_election.cc:341] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:44.233196 10090 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:44.233307 10164 consensus_queue.cc:1035] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:44.234328 10130 raft_consensus.cc:348] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.235008 10130 raft_consensus.cc:374] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.235253 10130 raft_consensus.cc:729] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
W20240426 09:54:44.235560 10055 leader_election.cc:341] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:44.235963 10055 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:44.235884 10130 consensus_queue.cc:260] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.237725 10130 ts_tablet_manager.cc:1398] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240426 09:54:44.238507 10130 tablet_bootstrap.cc:492] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:44.240921 10211 raft_consensus.cc:2726] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:44.228034 10131 tablet_bootstrap.cc:492] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:44.247145 10131 tablet_bootstrap.cc:654] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.270395 10130 tablet_bootstrap.cc:654] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.273857 9838 catalog_manager.cc:5484] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:44.274901 10200 consensus_queue.cc:1035] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 09:54:44.275686 10206 consensus_queue.cc:1035] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 09:54:44.276361 9838 catalog_manager.cc:5484] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:44.280273 10185 consensus_queue.cc:1035] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 09:54:44.290087 9940 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
W20240426 09:54:44.324661 9904 consensus_peers.cc:485] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 -> Peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Couldn't send request to peer d3a480e831d649139ff016fb68aee3ee. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:44.337342 10245 consensus_queue.cc:1035] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240426 09:54:44.367313 10211 consensus_queue.cc:1035] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:44.373371 10206 consensus_queue.cc:1035] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:44.386197 10185 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:44.386687 10185 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.388626 10185 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:44.391693 10130 tablet_bootstrap.cc:492] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:44.392140 10130 ts_tablet_manager.cc:1367] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.154s user 0.011s sys 0.000s
I20240426 09:54:44.394389 10130 raft_consensus.cc:348] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.397217 10130 raft_consensus.cc:374] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.400847 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:44.397464 10130 raft_consensus.cc:729] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.403578 10130 consensus_queue.cc:260] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.383069 10211 consensus_queue.cc:1035] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20240426 09:54:44.406917 10132 tablet_bootstrap.cc:492] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:44.407481 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:44.408100 10087 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:44.409819 10206 consensus_queue.cc:1035] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:44.406921 9939 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:44.413969 9982 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:44.407501 10132 ts_tablet_manager.cc:1367] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.206s user 0.012s sys 0.003s
I20240426 09:54:44.415040 10245 consensus_queue.cc:1035] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:44.416723 10245 raft_consensus.cc:2781] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:44.417038 10245 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:44.417369 10245 raft_consensus.cc:3037] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:44.417106 10132 raft_consensus.cc:348] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.417800 10132 raft_consensus.cc:374] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.418089 10132 raft_consensus.cc:729] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.418691 10132 consensus_queue.cc:260] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.397626 9839 catalog_manager.cc:5484] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:44.425168 10132 ts_tablet_manager.cc:1398] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240426 09:54:44.425786 10132 tablet_bootstrap.cc:492] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:44.427788 10131 tablet_bootstrap.cc:492] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:44.428206 10131 ts_tablet_manager.cc:1367] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.200s user 0.013s sys 0.016s
I20240426 09:54:44.430567 10131 raft_consensus.cc:348] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.431180 10131 raft_consensus.cc:374] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.431427 10131 raft_consensus.cc:729] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.432003 10131 consensus_queue.cc:260] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.434075 10131 ts_tablet_manager.cc:1398] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 09:54:44.436628 10131 tablet_bootstrap.cc:492] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:44.437340 10130 ts_tablet_manager.cc:1398] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.045s user 0.007s sys 0.000s
I20240426 09:54:44.438243 10130 tablet_bootstrap.cc:492] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:44.445133 10185 consensus_queue.cc:1035] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:44.454123 10130 tablet_bootstrap.cc:654] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.434387 10245 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.466284 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:44.469226 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:44.469794 10090 raft_consensus.cc:3037] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:44.476454 10131 tablet_bootstrap.cc:654] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.469214 9938 raft_consensus.cc:3037] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:44.467568 10245 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:44.506953 10185 consensus_queue.cc:1035] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:44.514503 10132 tablet_bootstrap.cc:654] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.517210 10131 tablet_bootstrap.cc:492] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:44.520655 10131 ts_tablet_manager.cc:1367] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.086s user 0.028s sys 0.003s
I20240426 09:54:44.523648 10131 raft_consensus.cc:348] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.524941 10131 raft_consensus.cc:374] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.525565 10131 raft_consensus.cc:729] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.520022 9938 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:44.527148 9982 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:44.527395 10131 consensus_queue.cc:260] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.528578 10090 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:44.529052 10185 raft_consensus.cc:2781] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:44.529537 10185 raft_consensus.cc:686] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:44.530407 10130 tablet_bootstrap.cc:492] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:44.530284 10185 consensus_queue.cc:237] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.530853 10130 ts_tablet_manager.cc:1367] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.093s user 0.008s sys 0.005s
I20240426 09:54:44.531404 10131 ts_tablet_manager.cc:1398] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240426 09:54:44.532231 10131 tablet_bootstrap.cc:492] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:44.532708 10132 tablet_bootstrap.cc:492] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:44.533151 10132 ts_tablet_manager.cc:1367] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.107s user 0.011s sys 0.001s
I20240426 09:54:44.533980 10130 raft_consensus.cc:348] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.535341 10130 raft_consensus.cc:374] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.535596 10130 raft_consensus.cc:729] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.536149 10130 consensus_queue.cc:260] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.537007 10132 raft_consensus.cc:348] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.538197 10130 ts_tablet_manager.cc:1398] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240426 09:54:44.544824 10132 raft_consensus.cc:374] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.545171 10132 raft_consensus.cc:729] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.545457 10130 tablet_bootstrap.cc:492] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:44.546074 10132 consensus_queue.cc:260] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.548475 10132 ts_tablet_manager.cc:1398] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20240426 09:54:44.549397 10132 tablet_bootstrap.cc:492] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:44.550664 10130 tablet_bootstrap.cc:654] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.542487 10131 tablet_bootstrap.cc:654] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.558485 10132 tablet_bootstrap.cc:654] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.580332 10131 tablet_bootstrap.cc:492] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:44.580762 10131 ts_tablet_manager.cc:1367] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.009s
I20240426 09:54:44.583261 10131 raft_consensus.cc:348] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.583976 10131 raft_consensus.cc:374] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.584223 10131 raft_consensus.cc:729] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.584792 10131 consensus_queue.cc:260] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.586886 9837 catalog_manager.cc:5484] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:44.588405 10131 ts_tablet_manager.cc:1398] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240426 09:54:44.589226 10131 tablet_bootstrap.cc:492] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:44.589351 10132 tablet_bootstrap.cc:492] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:44.589794 10132 ts_tablet_manager.cc:1367] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.009s
I20240426 09:54:44.593835 10164 consensus_queue.cc:1035] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:44.595248 10131 tablet_bootstrap.cc:654] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.597990 10130 tablet_bootstrap.cc:492] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:44.598340 10130 ts_tablet_manager.cc:1367] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.053s user 0.013s sys 0.004s
I20240426 09:54:44.606142 10132 raft_consensus.cc:348] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.610272 10132 raft_consensus.cc:374] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.610548 10132 raft_consensus.cc:729] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.612169 10132 consensus_queue.cc:260] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.608341 10130 raft_consensus.cc:348] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.613098 10130 raft_consensus.cc:374] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.613390 10130 raft_consensus.cc:729] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.614691 10130 consensus_queue.cc:260] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.617693 10132 ts_tablet_manager.cc:1398] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20240426 09:54:44.644416 10132 tablet_bootstrap.cc:492] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:44.649008 10130 ts_tablet_manager.cc:1398] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.050s user 0.003s sys 0.003s
I20240426 09:54:44.650012 10130 tablet_bootstrap.cc:492] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:44.660491 10162 consensus_queue.cc:1035] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20240426 09:54:44.668968 10132 tablet_bootstrap.cc:654] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.670827 10131 tablet_bootstrap.cc:492] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:44.655215 10130 tablet_bootstrap.cc:654] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.682420 10131 ts_tablet_manager.cc:1367] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.093s user 0.044s sys 0.026s
I20240426 09:54:44.684721 10131 raft_consensus.cc:348] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.685382 10131 raft_consensus.cc:374] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.685631 10131 raft_consensus.cc:729] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.686174 10131 consensus_queue.cc:260] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.688292 10131 ts_tablet_manager.cc:1398] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20240426 09:54:44.689071 10131 tablet_bootstrap.cc:492] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:44.693673 10131 tablet_bootstrap.cc:654] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.694892 10132 tablet_bootstrap.cc:492] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:44.696096 10132 ts_tablet_manager.cc:1367] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.052s user 0.014s sys 0.000s
I20240426 09:54:44.699066 10132 raft_consensus.cc:348] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.700192 10132 raft_consensus.cc:374] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.701375 10130 tablet_bootstrap.cc:492] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:44.701625 10132 raft_consensus.cc:729] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.701843 10130 ts_tablet_manager.cc:1367] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.012s
I20240426 09:54:44.702327 10132 consensus_queue.cc:260] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.702905 10131 tablet_bootstrap.cc:492] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:44.703373 10131 ts_tablet_manager.cc:1367] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20240426 09:54:44.704355 10132 ts_tablet_manager.cc:1398] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240426 09:54:44.704643 10130 raft_consensus.cc:348] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.705322 10132 tablet_bootstrap.cc:492] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:44.705526 10130 raft_consensus.cc:374] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.705924 10130 raft_consensus.cc:729] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.705471 10131 raft_consensus.cc:348] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.706301 10131 raft_consensus.cc:374] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.706560 10131 raft_consensus.cc:729] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.706588 10130 consensus_queue.cc:260] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.707238 10131 consensus_queue.cc:260] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.708590 10130 ts_tablet_manager.cc:1398] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20240426 09:54:44.709470 10130 tablet_bootstrap.cc:492] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:44.711728 10131 ts_tablet_manager.cc:1398] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240426 09:54:44.712546 10131 tablet_bootstrap.cc:492] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:44.716701 10131 tablet_bootstrap.cc:654] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.718152 10132 tablet_bootstrap.cc:654] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.726006 10131 tablet_bootstrap.cc:492] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:44.726331 10131 ts_tablet_manager.cc:1367] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.002s
I20240426 09:54:44.728657 10131 raft_consensus.cc:348] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.729310 10131 raft_consensus.cc:374] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.729326 10130 tablet_bootstrap.cc:654] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.729663 10131 raft_consensus.cc:729] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.730556 10132 tablet_bootstrap.cc:492] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:44.730357 10131 consensus_queue.cc:260] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.730976 10132 ts_tablet_manager.cc:1367] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.006s
I20240426 09:54:44.731174 10211 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:44.731609 10211 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.733093 10132 raft_consensus.cc:348] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.733462 10211 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:44.733739 10132 raft_consensus.cc:374] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.734045 10132 raft_consensus.cc:729] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.735425 10131 ts_tablet_manager.cc:1398] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240426 09:54:44.735525 10132 consensus_queue.cc:260] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.736316 10131 tablet_bootstrap.cc:492] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:44.737646 10132 ts_tablet_manager.cc:1398] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20240426 09:54:44.738528 10132 tablet_bootstrap.cc:492] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:44.739688 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:44.738530 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:44.741421 10131 tablet_bootstrap.cc:654] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.741839 10014 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
W20240426 09:54:44.743566 10056 leader_election.cc:341] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:44.743952 10132 tablet_bootstrap.cc:654] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.744652 10185 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:44.745136 10185 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.750882 10055 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:44.752746 10211 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:44.753156 10211 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:44.753190 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:44.753628 10211 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:44.754312 10185 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:54:44.755009 9982 leader_election.cc:341] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:44.757927 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:44.758538 10088 raft_consensus.cc:2454] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:44.759617 9979 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:44.760598 10185 raft_consensus.cc:2781] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:44.760922 10185 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:44.761204 10185 raft_consensus.cc:3037] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:44.762487 10132 tablet_bootstrap.cc:492] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:44.762936 10132 ts_tablet_manager.cc:1367] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.011s
I20240426 09:54:44.766328 10132 raft_consensus.cc:348] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.766995 10132 raft_consensus.cc:374] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.767364 10132 raft_consensus.cc:729] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.767069 10185 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.768043 10132 consensus_queue.cc:260] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.768998 10131 tablet_bootstrap.cc:492] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:44.769403 10131 ts_tablet_manager.cc:1367] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.000s
I20240426 09:54:44.770318 10132 ts_tablet_manager.cc:1398] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240426 09:54:44.771200 10132 tablet_bootstrap.cc:492] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:44.771108 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:44.771442 10164 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:44.771687 10088 raft_consensus.cc:3037] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:44.771803 10131 raft_consensus.cc:348] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.772504 10131 raft_consensus.cc:374] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.772768 10131 raft_consensus.cc:729] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.773336 10131 consensus_queue.cc:260] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.775370 10131 ts_tablet_manager.cc:1398] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240426 09:54:44.776149 10131 tablet_bootstrap.cc:492] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:44.777175 10132 tablet_bootstrap.cc:654] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.777832 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:54:44.779817 9982 leader_election.cc:341] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:44.779258 10088 raft_consensus.cc:2454] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:44.780951 10185 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:44.782119 10131 tablet_bootstrap.cc:654] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.783098 9979 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:44.784137 10245 raft_consensus.cc:2781] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:44.784551 10245 raft_consensus.cc:686] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:44.785845 10245 consensus_queue.cc:237] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.796661 10130 tablet_bootstrap.cc:492] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:44.797060 10130 ts_tablet_manager.cc:1367] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.088s user 0.011s sys 0.015s
I20240426 09:54:44.771893 10164 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.799544 10164 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:44.799814 10130 raft_consensus.cc:348] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.800480 10130 raft_consensus.cc:374] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.800595 10087 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:44.800767 10130 raft_consensus.cc:729] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.801479 10130 consensus_queue.cc:260] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.803802 10130 ts_tablet_manager.cc:1398] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240426 09:54:44.804968 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:44.806648 10130 tablet_bootstrap.cc:492] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:44.811728 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:44.813098 10055 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:44.815023 10130 tablet_bootstrap.cc:654] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.826433 10130 tablet_bootstrap.cc:492] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:44.826829 10130 ts_tablet_manager.cc:1367] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.001s
I20240426 09:54:44.828922 10130 raft_consensus.cc:348] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.829550 10130 raft_consensus.cc:374] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:44.829811 10130 raft_consensus.cc:729] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:44.830503 10130 consensus_queue.cc:260] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.832664 10130 ts_tablet_manager.cc:1398] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240426 09:54:44.833410 10130 tablet_bootstrap.cc:492] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:44.853402 9838 catalog_manager.cc:5484] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:44.862475 10245 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:44.875339 10245 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.877995 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:54:44.879384 9982 leader_election.cc:341] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:44.880538 10245 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:44.881170 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:54:44.882367 9979 leader_election.cc:341] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:44.882812 9979 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:44.883790 10245 raft_consensus.cc:2726] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:44.856002 10211 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.868623 10206 raft_consensus.cc:2726] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:44.894165 10178 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:44.894626 10178 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.896487 10178 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:44.898944 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:44.899531 10014 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:44.900653 9906 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:44.902215 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:44.902806 10088 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:44.902885 10178 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:44.903728 10178 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:44.904042 10178 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:44.930075 10245 consensus_queue.cc:1035] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:44.937924 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:44.939085 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:54:44.939224 10056 leader_election.cc:341] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:44.939679 10014 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:44.952067 10130 tablet_bootstrap.cc:654] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:44.959050 10211 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:44.948678 10178 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.974833 10178 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:44.980314 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:44.980903 10016 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:44.981375 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:44.985699 10088 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:44.984643 10185 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:44.990003 10088 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:44.990530 10185 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.991263 9904 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:44.992345 10185 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:44.992944 10211 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:44.992980 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:44.993685 9940 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:44.993525 10211 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:44.994722 9982 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:44.996752 10185 raft_consensus.cc:2781] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:44.997066 10185 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:44.997360 10185 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:45.003566 10185 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.005987 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:45.006520 9940 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:45.007655 10014 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:45.006508 10211 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:45.010522 10193 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:45.011034 10193 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.013509 10193 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:45.014245 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:45.015054 10014 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:45.016355 9906 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:45.016793 9940 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:45.017745 9982 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:45.018481 10184 raft_consensus.cc:2781] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:45.018959 10184 raft_consensus.cc:686] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:45.019163 10139 raft_consensus.cc:2781] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:45.019560 10139 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:44.995044 10178 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:45.019863 10139 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:45.021528 10184 consensus_queue.cc:237] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.018864 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:45.033217 10185 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:45.034350 10292 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:45.034794 10292 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.036537 10292 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:45.025759 10178 raft_consensus.cc:686] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:54:45.063031 10088 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:45.088003 10137 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:45.063838 10178 consensus_queue.cc:237] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.301434 10137 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.304342 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:45.129909 10206 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:45.308296 10137 leader_election.cc:288] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:45.309273 10016 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:45.310047 10137 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:45.310494 10137 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.312536 10137 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:45.313436 10132 tablet_bootstrap.cc:492] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:45.313912 10132 ts_tablet_manager.cc:1367] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.543s user 0.085s sys 0.065s
I20240426 09:54:45.315665 10130 tablet_bootstrap.cc:492] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:45.316160 10130 ts_tablet_manager.cc:1367] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.483s user 0.006s sys 0.021s
I20240426 09:54:45.316365 10132 raft_consensus.cc:348] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.317096 10132 raft_consensus.cc:374] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:45.317363 10132 raft_consensus.cc:729] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:45.307256 10206 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.368072 10131 tablet_bootstrap.cc:492] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:45.331013 10132 consensus_queue.cc:260] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.384410 10131 ts_tablet_manager.cc:1367] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.608s user 0.110s sys 0.106s
I20240426 09:54:45.318694 10130 raft_consensus.cc:348] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.387131 10130 raft_consensus.cc:374] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:45.387418 10130 raft_consensus.cc:729] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:45.387830 10130 consensus_queue.cc:260] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.389823 10132 ts_tablet_manager.cc:1398] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.076s user 0.010s sys 0.007s
W20240426 09:54:45.378166 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 312819us
I20240426 09:54:45.386464 10300 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:45.392449 10206 leader_election.cc:288] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:45.391120 10132 tablet_bootstrap.cc:492] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:45.392540 10300 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.394865 10300 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:45.143697 10139 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.388376 10193 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:45.396333 10193 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.398015 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:45.398212 10193 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:45.399941 10139 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:54:45.401854 9887 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 09:54:45.151909 10055 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:45.404114 10206 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:45.404611 10206 raft_consensus.cc:686] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:45.405397 10206 consensus_queue.cc:237] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.412600 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:45.414505 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:45.414433 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:45.415139 10089 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:45.415469 10090 raft_consensus.cc:2454] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:45.415297 10131 raft_consensus.cc:348] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.415997 10131 raft_consensus.cc:374] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:45.416281 10131 raft_consensus.cc:729] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:45.419023 10131 consensus_queue.cc:260] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:45.410979 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 259222us
I20240426 09:54:45.414912 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:45.423533 10130 ts_tablet_manager.cc:1398] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.107s user 0.017s sys 0.005s
I20240426 09:54:45.424504 10131 ts_tablet_manager.cc:1398] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.040s user 0.005s sys 0.000s
I20240426 09:54:45.424691 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:45.425199 10016 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:45.425345 10131 tablet_bootstrap.cc:492] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:45.428719 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:45.432256 9938 raft_consensus.cc:2454] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:45.433791 10087 raft_consensus.cc:2379] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:54:45.435513 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:45.436528 9940 raft_consensus.cc:2379] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 1: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:54:45.437834 9837 catalog_manager.cc:5484] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:45.438930 10131 tablet_bootstrap.cc:654] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:45.438799 10016 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:45.440361 10130 tablet_bootstrap.cc:492] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:45.439607 9838 catalog_manager.cc:5484] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:45.441308 10089 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:45.443418 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:45.445858 10090 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:45.439839 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:45.447368 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:54:45.435734 9906 leader_election.cc:341] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:45.450681 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:45.451371 10088 raft_consensus.cc:2454] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:45.452401 10090 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:45.454123 9906 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:45.458115 10303 raft_consensus.cc:2781] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:45.458601 10303 raft_consensus.cc:686] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:54:45.459565 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:45.459391 10303 consensus_queue.cc:237] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.488358 9939 raft_consensus.cc:3037] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:45.498443 10184 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:45.562794 10184 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.633047 10184 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:45.633591 10184 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:45.634016 10184 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.644107 10185 consensus_queue.cc:1035] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:45.645956 10132 tablet_bootstrap.cc:654] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:45.647845 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:45.650104 10184 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:45.650185 9979 leader_election.cc:302] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:45.651110 10184 raft_consensus.cc:2781] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:45.651494 10184 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:45.651850 10184 raft_consensus.cc:3037] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:45.646559 10056 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:45.657081 10211 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:54:45.657171 10055 leader_election.cc:341] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:45.657601 10211 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.660672 10297 raft_consensus.cc:2781] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:45.661013 10297 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:45.661345 10130 tablet_bootstrap.cc:654] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:45.661326 10297 raft_consensus.cc:3037] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:45.665832 9904 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:54:45.667481 10300 raft_consensus.cc:2781] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240426 09:54:45.662760 9982 leader_election.cc:341] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:45.668639 9904 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:45.669392 10193 raft_consensus.cc:2726] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:45.670271 10211 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:45.668113 10300 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:45.672094 10300 raft_consensus.cc:3037] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:45.686923 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:45.697206 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:45.698578 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:45.699745 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:45.701017 10300 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.703253 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:45.703184 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:45.732343 10193 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:45.732782 10193 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.734511 10193 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:45.728677 10185 consensus_queue.cc:1035] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240426 09:54:45.728471 10089 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:45.726001 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:45.740545 10132 tablet_bootstrap.cc:492] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:45.740968 10132 ts_tablet_manager.cc:1367] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.350s user 0.016s sys 0.076s
I20240426 09:54:45.729384 10300 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:45.745051 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:45.745476 10130 tablet_bootstrap.cc:492] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:45.745882 10130 ts_tablet_manager.cc:1367] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.306s user 0.012s sys 0.019s
I20240426 09:54:45.745725 10055 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:45.745975 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:45.746943 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:45.747221 10292 raft_consensus.cc:2726] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:45.748373 10130 raft_consensus.cc:348] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.748999 10130 raft_consensus.cc:374] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:45.749190 10130 raft_consensus.cc:729] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:45.748785 10297 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.749629 10130 consensus_queue.cc:260] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.751384 10130 ts_tablet_manager.cc:1398] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240426 09:54:45.752067 10130 tablet_bootstrap.cc:492] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:45.752812 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:45.753993 10089 raft_consensus.cc:2379] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
W20240426 09:54:45.744357 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 44642.8us
I20240426 09:54:45.754287 10184 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:45.755383 10056 leader_election.cc:341] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
W20240426 09:54:45.751271 9982 leader_election.cc:341] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:45.745678 10012 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
W20240426 09:54:45.756450 9982 leader_election.cc:341] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:45.756958 10016 raft_consensus.cc:2379] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 1: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
W20240426 09:54:45.757292 9982 leader_election.cc:341] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:45.757505 10184 leader_election.cc:288] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:45.759740 10131 tablet_bootstrap.cc:492] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:45.760210 10131 ts_tablet_manager.cc:1367] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.335s user 0.022s sys 0.011s
I20240426 09:54:45.761015 10132 raft_consensus.cc:348] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.761730 10132 raft_consensus.cc:374] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:45.762387 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:45.763562 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:45.764828 9941 raft_consensus.cc:2379] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 1: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:54:45.762012 10132 raft_consensus.cc:729] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:45.766875 10132 consensus_queue.cc:260] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.768824 10132 ts_tablet_manager.cc:1398] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.028s user 0.003s sys 0.002s
I20240426 09:54:45.769758 10132 tablet_bootstrap.cc:492] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:45.750643 10297 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:54:45.773723 10055 leader_election.cc:341] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
W20240426 09:54:45.772050 9979 leader_election.cc:341] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:45.775172 9979 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:45.775856 10185 raft_consensus.cc:2726] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240426 09:54:45.777127 9979 leader_election.cc:341] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:45.777055 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:45.777711 9939 raft_consensus.cc:3037] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:45.773568 10309 consensus_queue.cc:1035] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240426 09:54:45.773222 10056 leader_election.cc:341] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:45.777603 9979 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:45.780033 10130 tablet_bootstrap.cc:654] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:45.780177 10131 raft_consensus.cc:348] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.784161 10131 raft_consensus.cc:374] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:45.784371 10131 raft_consensus.cc:729] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:45.784801 10131 consensus_queue.cc:260] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.783464 10056 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:45.787379 10131 ts_tablet_manager.cc:1398] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.027s user 0.003s sys 0.002s
I20240426 09:54:45.786408 10297 raft_consensus.cc:2726] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240426 09:54:45.788765 10131 tablet_bootstrap.cc:492] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:45.788283 10309 raft_consensus.cc:2726] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:45.791656 10055 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:45.783427 9939 raft_consensus.cc:2454] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:45.792976 10297 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:45.794462 10132 tablet_bootstrap.cc:654] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:45.794082 10297 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:45.795161 10297 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:45.795034 10055 leader_election.cc:302] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:45.795939 9982 leader_election.cc:302] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:45.797632 10137 raft_consensus.cc:2781] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:45.799113 10137 raft_consensus.cc:686] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:45.799990 10137 consensus_queue.cc:237] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.800452 9979 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:45.798851 10292 raft_consensus.cc:3037] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:45.805291 10309 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:45.812510 10131 tablet_bootstrap.cc:654] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:45.809010 10309 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:45.814787 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:45.815362 10309 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:45.826449 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:45.827203 10013 raft_consensus.cc:2379] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:54:45.827790 10300 consensus_queue.cc:1035] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:45.827404 9839 catalog_manager.cc:5484] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
W20240426 09:54:45.839871 9906 leader_election.cc:341] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:45.840281 9906 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:45.842474 10300 raft_consensus.cc:2726] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240426 09:54:45.842803 10303 consensus_queue.cc:1035] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:45.847146 9906 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:45.850342 10303 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:45.856220 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:45.860191 10309 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.864281 10292 raft_consensus.cc:2726] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:45.850466 10297 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.866940 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:54:45.868124 9982 leader_election.cc:341] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:45.871133 10130 tablet_bootstrap.cc:492] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:45.871512 10130 ts_tablet_manager.cc:1367] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.120s user 0.014s sys 0.015s
I20240426 09:54:45.889549 10309 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:45.890990 10130 raft_consensus.cc:348] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.891757 10130 raft_consensus.cc:374] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:45.891994 10130 raft_consensus.cc:729] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:45.895467 10132 tablet_bootstrap.cc:492] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:45.895946 10132 ts_tablet_manager.cc:1367] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.126s user 0.020s sys 0.016s
I20240426 09:54:45.895860 10130 consensus_queue.cc:260] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.921649 10130 ts_tablet_manager.cc:1398] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.050s user 0.004s sys 0.003s
I20240426 09:54:45.922550 10130 tablet_bootstrap.cc:492] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:45.917703 9837 catalog_manager.cc:5484] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:45.936224 10297 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:45.937860 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:45.938144 10089 raft_consensus.cc:2454] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:45.944053 10309 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:45.944510 10309 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:45.945096 10056 leader_election.cc:341] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:45.946352 10309 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:45.947633 10132 raft_consensus.cc:348] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.948299 10132 raft_consensus.cc:374] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:45.948558 10132 raft_consensus.cc:729] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:45.949128 10132 consensus_queue.cc:260] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:45.949728 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:45.911397 10303 raft_consensus.cc:2726] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:45.957062 10132 ts_tablet_manager.cc:1398] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.061s user 0.005s sys 0.000s
I20240426 09:54:45.958904 10132 tablet_bootstrap.cc:492] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:45.951843 10185 consensus_queue.cc:1035] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240426 09:54:45.961107 10088 raft_consensus.cc:2379] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:54:45.960644 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:45.963222 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:45.963936 10012 raft_consensus.cc:2379] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 1: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:54:45.985388 10137 consensus_queue.cc:1035] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240426 09:54:45.998929 10132 tablet_bootstrap.cc:654] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:45.999965 10130 tablet_bootstrap.cc:654] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
W20240426 09:54:45.994798 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 44774.8us
I20240426 09:54:46.003743 9979 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:46.004482 10137 raft_consensus.cc:2726] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240426 09:54:46.005658 9982 leader_election.cc:341] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.006958 10131 tablet_bootstrap.cc:492] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:46.007426 10131 ts_tablet_manager.cc:1367] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.219s user 0.029s sys 0.019s
I20240426 09:54:46.006443 9837 catalog_manager.cc:5484] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:46.016301 10131 raft_consensus.cc:348] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.017323 10131 raft_consensus.cc:374] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.017525 10131 raft_consensus.cc:729] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.018020 10131 consensus_queue.cc:260] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.020901 10131 ts_tablet_manager.cc:1398] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240426 09:54:46.021639 10131 tablet_bootstrap.cc:492] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:46.021528 10055 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:46.023938 10297 raft_consensus.cc:2726] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240426 09:54:46.030360 10131 tablet_bootstrap.cc:654] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.035748 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:46.046165 10132 tablet_bootstrap.cc:492] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:46.046597 10132 ts_tablet_manager.cc:1367] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.088s user 0.004s sys 0.016s
I20240426 09:54:46.048769 10132 raft_consensus.cc:348] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.049454 10132 raft_consensus.cc:374] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.049718 10132 raft_consensus.cc:729] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.061913 10132 consensus_queue.cc:260] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.063838 10132 ts_tablet_manager.cc:1398] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.017s user 0.002s sys 0.003s
I20240426 09:54:46.064656 10132 tablet_bootstrap.cc:492] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:46.074268 10131 tablet_bootstrap.cc:492] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:46.074615 10131 ts_tablet_manager.cc:1367] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.053s user 0.019s sys 0.006s
I20240426 09:54:46.077348 10130 tablet_bootstrap.cc:492] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:46.077829 10130 ts_tablet_manager.cc:1367] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.155s user 0.009s sys 0.005s
W20240426 09:54:46.079934 9979 leader_election.cc:341] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.080386 9979 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:46.080184 10130 raft_consensus.cc:348] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.081029 10130 raft_consensus.cc:374] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.081350 10130 raft_consensus.cc:729] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.082154 10137 raft_consensus.cc:2726] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:46.081957 10130 consensus_queue.cc:260] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.084705 10130 ts_tablet_manager.cc:1398] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240426 09:54:46.084787 10132 tablet_bootstrap.cc:654] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.085769 10297 consensus_queue.cc:1035] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:46.085584 10130 tablet_bootstrap.cc:492] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:46.086093 10313 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.092932 10313 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.094959 10313 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:46.097759 10130 tablet_bootstrap.cc:654] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.087136 10131 raft_consensus.cc:348] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.100210 10131 raft_consensus.cc:374] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.100494 10131 raft_consensus.cc:729] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.101066 10131 consensus_queue.cc:260] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.101459 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:46.102082 10013 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:46.103101 10132 tablet_bootstrap.cc:492] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:46.103560 10132 ts_tablet_manager.cc:1367] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.039s user 0.011s sys 0.004s
I20240426 09:54:46.103418 10055 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:46.104197 10297 raft_consensus.cc:2781] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:46.104533 10297 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:46.104840 10297 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.106029 10132 raft_consensus.cc:348] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.106791 10132 raft_consensus.cc:374] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.107076 10132 raft_consensus.cc:729] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.107667 10132 consensus_queue.cc:260] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.109992 10132 ts_tablet_manager.cc:1398] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240426 09:54:46.109767 10313 consensus_queue.cc:1035] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:46.112545 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:46.113489 10132 tablet_bootstrap.cc:492] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:46.116091 10131 ts_tablet_manager.cc:1398] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20240426 09:54:46.116906 10131 tablet_bootstrap.cc:492] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:46.115072 10301 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240426 09:54:46.118824 10301 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:46.120950 10297 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:46.124886 10056 leader_election.cc:341] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.127307 10132 tablet_bootstrap.cc:654] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.135977 10292 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.136425 10292 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.138216 10292 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:46.138732 10297 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:46.138903 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:46.146472 10012 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.146705 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:54:46.152385 10055 leader_election.cc:341] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.161087 10012 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:46.162671 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:46.163475 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:54:46.165136 10056 leader_election.cc:341] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.169132 10055 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:46.169559 10056 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:46.171943 10313 raft_consensus.cc:2781] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
W20240426 09:54:46.172361 10056 leader_election.cc:341] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.172379 10313 raft_consensus.cc:686] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:46.175526 10313 consensus_queue.cc:237] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.173081 10298 raft_consensus.cc:2726] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:46.182619 10193 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.183091 10193 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.184338 10130 tablet_bootstrap.cc:492] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:46.184482 10131 tablet_bootstrap.cc:654] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.184770 10130 ts_tablet_manager.cc:1367] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.099s user 0.019s sys 0.005s
I20240426 09:54:46.187703 10193 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.189694 10130 raft_consensus.cc:348] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.190161 10130 raft_consensus.cc:374] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.190419 10130 raft_consensus.cc:729] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.191246 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:46.191123 10130 consensus_queue.cc:260] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.192402 10013 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:46.193360 10130 ts_tablet_manager.cc:1398] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20240426 09:54:46.193439 9906 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:46.194205 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:54:46.195444 9904 leader_election.cc:341] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.197247 10193 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:46.197638 10193 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:46.198004 10193 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.196960 10130 tablet_bootstrap.cc:492] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:46.208308 10193 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.215078 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:46.215628 10012 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.220057 9839 catalog_manager.cc:5484] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:46.220130 10130 tablet_bootstrap.cc:654] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.221398 10131 tablet_bootstrap.cc:492] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:46.221832 10131 ts_tablet_manager.cc:1367] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.105s user 0.010s sys 0.004s
I20240426 09:54:46.223942 10012 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:46.224117 10131 raft_consensus.cc:348] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.224860 10131 raft_consensus.cc:374] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.225108 10131 raft_consensus.cc:729] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.225059 9906 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:46.225972 10139 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:46.225888 10131 consensus_queue.cc:260] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.226455 10139 raft_consensus.cc:686] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:54:46.227388 10139 consensus_queue.cc:237] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.228452 10132 tablet_bootstrap.cc:492] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:46.228909 10132 ts_tablet_manager.cc:1367] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.117s user 0.015s sys 0.003s
I20240426 09:54:46.230633 10331 consensus_queue.cc:1035] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 09:54:46.231196 10132 raft_consensus.cc:348] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.231846 10132 raft_consensus.cc:374] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.232102 10132 raft_consensus.cc:729] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.232797 10132 consensus_queue.cc:260] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.236943 10132 ts_tablet_manager.cc:1398] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20240426 09:54:46.237831 10132 tablet_bootstrap.cc:492] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:46.243414 10131 ts_tablet_manager.cc:1398] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.021s user 0.007s sys 0.001s
I20240426 09:54:46.244333 10131 tablet_bootstrap.cc:492] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:46.249323 10131 tablet_bootstrap.cc:654] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.255970 10132 tablet_bootstrap.cc:654] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.277015 10193 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.280722 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:46.281406 10089 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.281684 10130 tablet_bootstrap.cc:492] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:46.282164 10130 ts_tablet_manager.cc:1367] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.085s user 0.024s sys 0.020s
I20240426 09:54:46.284499 10130 raft_consensus.cc:348] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.285162 10130 raft_consensus.cc:374] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.285409 10130 raft_consensus.cc:729] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.285949 10130 consensus_queue.cc:260] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.288229 10089 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:46.289531 10130 ts_tablet_manager.cc:1398] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20240426 09:54:46.290436 10130 tablet_bootstrap.cc:492] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:46.295454 10130 tablet_bootstrap.cc:654] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.307777 9837 catalog_manager.cc:5484] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:46.321105 10132 tablet_bootstrap.cc:492] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:46.321506 10132 ts_tablet_manager.cc:1367] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.084s user 0.013s sys 0.006s
I20240426 09:54:46.326517 10184 consensus_queue.cc:1035] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 09:54:46.334178 10130 tablet_bootstrap.cc:492] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:46.334619 10130 ts_tablet_manager.cc:1367] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.044s user 0.017s sys 0.005s
I20240426 09:54:46.337383 10132 raft_consensus.cc:348] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.338079 10132 raft_consensus.cc:374] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.338339 10132 raft_consensus.cc:729] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.335877 10131 tablet_bootstrap.cc:492] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:46.339784 10131 ts_tablet_manager.cc:1367] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.096s user 0.011s sys 0.014s
I20240426 09:54:46.342540 10131 raft_consensus.cc:348] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.343148 10131 raft_consensus.cc:374] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.343398 10131 raft_consensus.cc:729] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.338929 10132 consensus_queue.cc:260] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.348049 10132 ts_tablet_manager.cc:1398] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20240426 09:54:46.348870 10132 tablet_bootstrap.cc:492] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:46.352778 10139 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.353528 10132 tablet_bootstrap.cc:654] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.353205 10139 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.355326 10139 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.356668 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:46.357731 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:46.358296 10089 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:46.343961 10131 consensus_queue.cc:260] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.344235 10130 raft_consensus.cc:348] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.359283 10130 raft_consensus.cc:374] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.359594 10130 raft_consensus.cc:729] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.360183 10131 ts_tablet_manager.cc:1398] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.020s user 0.003s sys 0.002s
W20240426 09:54:46.360203 9906 leader_election.cc:341] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.361215 9904 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:54:46.364892 10139 raft_consensus.cc:2781] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:46.365231 10139 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:46.365535 10139 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.369742 10131 tablet_bootstrap.cc:492] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:46.371120 10193 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.371580 10193 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.373338 10193 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.373755 10193 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.374145 10193 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.375852 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:46.376441 10012 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:46.377856 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:46.378898 9906 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:46.379696 10303 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:46.380018 10303 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:46.380327 10303 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
W20240426 09:54:46.383500 9904 leader_election.cc:341] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.384692 10131 tablet_bootstrap.cc:654] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.388914 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:46.389086 10193 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.389637 10012 raft_consensus.cc:2379] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:54:46.390093 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:46.391883 9904 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:46.392942 10193 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.393388 10331 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.393815 10331 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.394971 10303 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.395656 10331 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.392925 10132 tablet_bootstrap.cc:492] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:46.396358 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:46.396803 10132 ts_tablet_manager.cc:1367] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.048s user 0.011s sys 0.007s
I20240426 09:54:46.397251 10303 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.360200 10130 consensus_queue.cc:260] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.398607 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:46.394490 10139 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.397791 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:46.402081 10139 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:54:46.399528 9982 leader_election.cc:341] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.400521 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
W20240426 09:54:46.404438 9979 leader_election.cc:341] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.404711 10131 tablet_bootstrap.cc:492] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:46.405144 10131 ts_tablet_manager.cc:1367] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.000s
I20240426 09:54:46.405691 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:46.406224 10089 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.407820 10131 raft_consensus.cc:348] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.408454 10131 raft_consensus.cc:374] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.408712 10131 raft_consensus.cc:729] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.409282 10131 consensus_queue.cc:260] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.411008 10331 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.399085 10132 raft_consensus.cc:348] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.411659 10132 raft_consensus.cc:374] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.411465 10331 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.411931 10132 raft_consensus.cc:729] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.412495 10089 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:46.412890 10132 consensus_queue.cc:260] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.413264 10331 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:54:46.415493 9904 leader_election.cc:341] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.416427 10131 ts_tablet_manager.cc:1398] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20240426 09:54:46.417265 10131 tablet_bootstrap.cc:492] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:46.401405 10130 ts_tablet_manager.cc:1398] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.066s user 0.005s sys 0.001s
I20240426 09:54:46.422565 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:46.423365 9940 raft_consensus.cc:2379] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 1: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:54:46.399205 10012 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.424206 10193 raft_consensus.cc:2726] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:46.424773 10130 tablet_bootstrap.cc:492] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:46.427394 10132 ts_tablet_manager.cc:1398] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20240426 09:54:46.428232 10132 tablet_bootstrap.cc:492] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:46.430235 10012 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:46.430696 10131 tablet_bootstrap.cc:654] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.422456 9979 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:46.432175 10184 raft_consensus.cc:2726] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:46.433491 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:46.434223 10089 raft_consensus.cc:2379] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:54:46.435467 9906 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:46.436334 10139 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:46.436578 10130 tablet_bootstrap.cc:654] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.437140 9904 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:46.439935 10193 raft_consensus.cc:2781] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:46.440366 10193 raft_consensus.cc:686] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:54:46.441299 10193 consensus_queue.cc:237] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.442800 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:46.443598 10131 tablet_bootstrap.cc:492] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:46.444116 10131 ts_tablet_manager.cc:1367] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.004s
I20240426 09:54:46.444752 10139 raft_consensus.cc:686] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:54:46.445513 10139 consensus_queue.cc:237] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.448184 10131 raft_consensus.cc:348] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.448931 10131 raft_consensus.cc:374] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.449270 10131 raft_consensus.cc:729] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.449381 9979 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:46.450285 10184 raft_consensus.cc:3037] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.450110 10131 consensus_queue.cc:260] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:46.453312 9906 leader_election.cc:341] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.456832 10184 raft_consensus.cc:2726] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:46.460117 10131 ts_tablet_manager.cc:1398] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20240426 09:54:46.461036 10131 tablet_bootstrap.cc:492] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:46.461637 10130 tablet_bootstrap.cc:492] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:46.464555 10130 ts_tablet_manager.cc:1367] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.045s user 0.025s sys 0.003s
I20240426 09:54:46.467998 10130 raft_consensus.cc:348] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.470883 10130 raft_consensus.cc:374] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.471201 10130 raft_consensus.cc:729] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.471810 10130 consensus_queue.cc:260] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.474550 10130 ts_tablet_manager.cc:1398] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20240426 09:54:46.475608 10130 tablet_bootstrap.cc:492] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:46.474900 10132 tablet_bootstrap.cc:654] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.472771 10131 tablet_bootstrap.cc:654] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.484535 10130 tablet_bootstrap.cc:654] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.504091 10130 tablet_bootstrap.cc:492] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:46.505074 10130 ts_tablet_manager.cc:1367] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.005s
I20240426 09:54:46.502662 9839 catalog_manager.cc:5484] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:46.508307 9839 catalog_manager.cc:5484] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:46.510371 10139 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.509019 10130 raft_consensus.cc:348] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.510887 10139 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.511638 10130 raft_consensus.cc:374] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.512228 10130 raft_consensus.cc:729] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.512766 10139 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.513442 10130 consensus_queue.cc:260] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.514369 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:46.515242 10088 raft_consensus.cc:2379] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:54:46.516898 10130 ts_tablet_manager.cc:1398] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20240426 09:54:46.518517 10130 tablet_bootstrap.cc:492] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
W20240426 09:54:46.517318 9887 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 09:54:46.520256 10131 tablet_bootstrap.cc:492] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:46.520713 10131 ts_tablet_manager.cc:1367] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.060s user 0.018s sys 0.007s
I20240426 09:54:46.523180 10131 raft_consensus.cc:348] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.523860 10131 raft_consensus.cc:374] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.524163 10131 raft_consensus.cc:729] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.524860 10131 consensus_queue.cc:260] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.525352 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:46.526134 10012 raft_consensus.cc:2379] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:54:46.527166 9906 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:46.534834 10132 tablet_bootstrap.cc:492] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:46.535190 10131 ts_tablet_manager.cc:1398] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20240426 09:54:46.525358 10130 tablet_bootstrap.cc:654] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.535248 10132 ts_tablet_manager.cc:1367] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.107s user 0.014s sys 0.022s
I20240426 09:54:46.537277 10139 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.539744 10132 raft_consensus.cc:348] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.540390 10132 raft_consensus.cc:374] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.540648 10132 raft_consensus.cc:729] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.541215 10132 consensus_queue.cc:260] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.543233 10132 ts_tablet_manager.cc:1398] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240426 09:54:46.543653 10139 raft_consensus.cc:2726] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:46.546211 10131 tablet_bootstrap.cc:492] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:46.547281 10132 tablet_bootstrap.cc:492] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:46.552129 10131 tablet_bootstrap.cc:654] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.552799 10132 tablet_bootstrap.cc:654] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.552821 10130 tablet_bootstrap.cc:492] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:46.553442 10130 ts_tablet_manager.cc:1367] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.005s
I20240426 09:54:46.555712 10130 raft_consensus.cc:348] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.556368 10130 raft_consensus.cc:374] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.556622 10130 raft_consensus.cc:729] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.557246 10130 consensus_queue.cc:260] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.559221 10130 ts_tablet_manager.cc:1398] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240426 09:54:46.559970 10130 tablet_bootstrap.cc:492] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:46.564679 10130 tablet_bootstrap.cc:654] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.567011 10184 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.567461 10184 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.569315 10184 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.572743 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:46.573442 10088 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:46.574508 10132 tablet_bootstrap.cc:492] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:46.574951 10132 ts_tablet_manager.cc:1367] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.005s
I20240426 09:54:46.575341 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:46.575978 9940 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:46.577179 10132 raft_consensus.cc:348] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.577935 10132 raft_consensus.cc:374] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.578227 10132 raft_consensus.cc:729] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.577891 9979 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:46.578934 10132 consensus_queue.cc:260] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.579967 10137 raft_consensus.cc:2781] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:46.580322 10137 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:46.580654 10137 raft_consensus.cc:3037] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.586730 10137 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.588814 10137 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.590354 10132 ts_tablet_manager.cc:1398] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20240426 09:54:46.590494 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:46.591234 10088 raft_consensus.cc:3037] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.595497 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:46.596100 9940 raft_consensus.cc:3037] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.596824 10131 tablet_bootstrap.cc:492] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:46.597298 10131 ts_tablet_manager.cc:1367] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.051s user 0.012s sys 0.023s
I20240426 09:54:46.599265 10088 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:46.599669 10131 raft_consensus.cc:348] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.600384 10131 raft_consensus.cc:374] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.600409 9979 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:46.600773 10131 raft_consensus.cc:729] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.601841 10132 tablet_bootstrap.cc:492] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:46.601569 10131 consensus_queue.cc:260] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.602025 10137 raft_consensus.cc:2781] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:46.604758 10137 raft_consensus.cc:686] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:46.605764 10137 consensus_queue.cc:237] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.608400 9940 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:46.609280 10132 tablet_bootstrap.cc:654] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.612092 10131 ts_tablet_manager.cc:1398] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20240426 09:54:46.614689 10130 tablet_bootstrap.cc:492] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:46.615125 10130 ts_tablet_manager.cc:1367] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.055s user 0.031s sys 0.020s
I20240426 09:54:46.617038 10131 tablet_bootstrap.cc:492] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:46.617403 10130 raft_consensus.cc:348] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.618089 10130 raft_consensus.cc:374] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.618359 10130 raft_consensus.cc:729] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.618942 10130 consensus_queue.cc:260] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.622115 10130 ts_tablet_manager.cc:1398] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240426 09:54:46.624995 10130 tablet_bootstrap.cc:492] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:46.627383 10131 tablet_bootstrap.cc:654] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.632012 10130 tablet_bootstrap.cc:654] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.633093 10132 tablet_bootstrap.cc:492] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:46.633491 10132 ts_tablet_manager.cc:1367] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.014s
I20240426 09:54:46.635373 10313 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.635803 10313 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.635833 9837 catalog_manager.cc:5484] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:46.637012 10132 raft_consensus.cc:348] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.637651 10132 raft_consensus.cc:374] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.637701 10313 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:46.637964 10132 raft_consensus.cc:729] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.638559 10132 consensus_queue.cc:260] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.640552 10132 ts_tablet_manager.cc:1398] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240426 09:54:46.641273 10132 tablet_bootstrap.cc:492] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:46.640841 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:46.643208 10012 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:46.643308 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:46.644635 10055 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
W20240426 09:54:46.646210 10056 leader_election.cc:341] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.648051 10297 raft_consensus.cc:2781] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:46.648428 10297 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:46.648757 10297 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.649032 10132 tablet_bootstrap.cc:654] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.653473 10130 tablet_bootstrap.cc:492] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:46.654986 10297 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.655512 10131 tablet_bootstrap.cc:492] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:46.656008 10131 ts_tablet_manager.cc:1367] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.007s
I20240426 09:54:46.653928 10130 ts_tablet_manager.cc:1367] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.000s
I20240426 09:54:46.657819 10297 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:46.658317 10297 consensus_queue.cc:1035] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:46.658442 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:46.658615 10130 raft_consensus.cc:348] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.659494 10130 raft_consensus.cc:374] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240426 09:54:46.659648 10056 leader_election.cc:341] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.661304 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:46.661840 10012 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.661877 10313 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.662376 10313 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.662727 10131 raft_consensus.cc:348] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.663412 10131 raft_consensus.cc:374] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.663671 10131 raft_consensus.cc:729] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.664216 10131 consensus_queue.cc:260] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.665128 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:46.665798 10016 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:46.659686 10130 raft_consensus.cc:729] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.666967 10055 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:46.667644 10130 consensus_queue.cc:260] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.667977 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:46.668581 10292 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:46.668542 9941 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:46.669001 10292 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:46.669343 10292 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.669955 10130 ts_tablet_manager.cc:1398] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240426 09:54:46.671983 10130 tablet_bootstrap.cc:492] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:46.676450 10131 ts_tablet_manager.cc:1398] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20240426 09:54:46.677471 10131 tablet_bootstrap.cc:492] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:46.680879 10130 tablet_bootstrap.cc:654] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.686607 10313 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:46.686848 10131 tablet_bootstrap.cc:654] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.683640 10292 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.689323 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:46.689644 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:46.690131 10313 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.690291 9941 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.690760 10016 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.691617 10130 tablet_bootstrap.cc:492] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:46.692047 10130 ts_tablet_manager.cc:1367] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.004s
I20240426 09:54:46.693675 10012 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:46.694298 10130 raft_consensus.cc:348] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.694733 10055 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:46.694973 10130 raft_consensus.cc:374] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.695366 10130 raft_consensus.cc:729] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.695518 10298 raft_consensus.cc:2781] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:46.696053 10298 raft_consensus.cc:686] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:46.696091 10130 consensus_queue.cc:260] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.697428 9941 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:46.697188 10298 consensus_queue.cc:237] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.698374 10016 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:46.699055 10130 ts_tablet_manager.cc:1398] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240426 09:54:46.699258 10292 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:46.699440 10055 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:46.690558 10313 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.700464 10297 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:46.701332 10313 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:46.701469 10130 tablet_bootstrap.cc:492] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:46.702198 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:46.703184 10297 raft_consensus.cc:686] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:46.703732 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:46.704345 10016 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:46.704041 10297 consensus_queue.cc:237] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:46.703478 10056 leader_election.cc:341] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.709126 10130 tablet_bootstrap.cc:654] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.712306 10132 tablet_bootstrap.cc:492] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:46.712735 10132 ts_tablet_manager.cc:1367] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.072s user 0.011s sys 0.004s
I20240426 09:54:46.714736 10055 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:46.715387 10132 raft_consensus.cc:348] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.715857 10292 raft_consensus.cc:2781] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:46.716257 10292 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:46.716603 10292 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.717674 10132 raft_consensus.cc:374] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.717952 10132 raft_consensus.cc:729] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.718782 10131 tablet_bootstrap.cc:492] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:46.718541 10132 consensus_queue.cc:260] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.719192 10131 ts_tablet_manager.cc:1367] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.004s
I20240426 09:54:46.720754 10132 ts_tablet_manager.cc:1398] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240426 09:54:46.721913 10132 tablet_bootstrap.cc:492] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:46.723712 10131 raft_consensus.cc:348] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.724359 10131 raft_consensus.cc:374] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.724601 10131 raft_consensus.cc:729] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.725106 10131 consensus_queue.cc:260] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.725963 10184 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.726384 10184 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.726909 10132 tablet_bootstrap.cc:654] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.727039 10131 ts_tablet_manager.cc:1398] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240426 09:54:46.727901 10131 tablet_bootstrap.cc:492] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:46.733206 10184 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.734117 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:46.736318 10131 tablet_bootstrap.cc:654] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.736598 10211 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.736665 10313 consensus_queue.cc:1035] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:46.738965 10211 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.741042 10211 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:54:46.741459 9982 leader_election.cc:341] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.742105 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:46.742342 10139 consensus_queue.cc:1035] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:46.742727 10088 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:46.743064 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:46.744756 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:46.745433 10013 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:46.746685 9979 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
W20240426 09:54:46.747227 10056 leader_election.cc:341] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.747519 10297 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.748034 10297 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.749732 10184 raft_consensus.cc:2781] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:46.750016 10297 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:46.750862 10055 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:46.751791 10139 consensus_queue.cc:1035] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:46.752211 10297 raft_consensus.cc:2781] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:46.752607 10297 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:46.752951 10297 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.754961 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:54:46.756596 10055 leader_election.cc:341] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.750093 10184 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:46.757705 10184 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.764755 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:54:46.766050 10056 leader_election.cc:341] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.766471 10056 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:46.767208 10211 raft_consensus.cc:2726] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:46.770982 10292 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.773341 10292 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:46.776525 10297 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.779199 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:46.779361 9838 catalog_manager.cc:5484] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:46.780619 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:46.785490 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:46.787317 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:54:46.791023 10056 leader_election.cc:341] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.791074 10013 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
W20240426 09:54:46.792479 10056 leader_election.cc:341] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.799746 10297 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:46.799531 10013 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:46.805267 10055 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:46.808584 10292 raft_consensus.cc:2781] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:46.809015 10292 raft_consensus.cc:686] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:46.809752 10292 consensus_queue.cc:237] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.802075 10184 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.818488 10016 raft_consensus.cc:2379] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 1: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:54:46.823328 10055 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:46.824923 10211 raft_consensus.cc:2726] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240426 09:54:46.830015 10139 consensus_queue.cc:1035] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:46.837271 10184 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.837476 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:54:46.839270 9982 leader_election.cc:341] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.841944 10130 tablet_bootstrap.cc:492] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:46.842396 10130 ts_tablet_manager.cc:1367] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.141s user 0.031s sys 0.017s
I20240426 09:54:46.839231 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:46.843551 10088 raft_consensus.cc:2379] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:54:46.846377 9979 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:46.848089 10331 raft_consensus.cc:2726] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240426 09:54:46.847905 10130 raft_consensus.cc:348] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.848609 10130 raft_consensus.cc:374] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.848924 10130 raft_consensus.cc:729] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.849655 10130 consensus_queue.cc:260] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.848631 10132 tablet_bootstrap.cc:492] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:46.851089 10132 ts_tablet_manager.cc:1367] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.129s user 0.012s sys 0.028s
I20240426 09:54:46.851881 10130 ts_tablet_manager.cc:1398] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240426 09:54:46.852731 10130 tablet_bootstrap.cc:492] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:46.853322 10132 raft_consensus.cc:348] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.853981 10132 raft_consensus.cc:374] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.854243 10132 raft_consensus.cc:729] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.858608 10131 tablet_bootstrap.cc:492] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:46.858731 10130 tablet_bootstrap.cc:654] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.859089 10131 ts_tablet_manager.cc:1367] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.131s user 0.004s sys 0.041s
I20240426 09:54:46.854629 9837 catalog_manager.cc:5484] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:46.861438 10131 raft_consensus.cc:348] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.861949 10139 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.862526 10139 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.862095 10131 raft_consensus.cc:374] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.868162 10131 raft_consensus.cc:729] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.868842 10131 consensus_queue.cc:260] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.871964 10131 ts_tablet_manager.cc:1398] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20240426 09:54:46.873558 10131 tablet_bootstrap.cc:492] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:46.874346 10139 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.871868 10012 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.876701 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:46.877281 10016 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:46.878520 9906 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:46.875105 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:46.879715 10303 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:46.880080 10303 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:46.880412 10303 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
W20240426 09:54:46.880867 9904 leader_election.cc:341] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.884945 10193 consensus_queue.cc:1035] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:46.888988 10303 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.891319 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:46.891865 10016 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.892115 10303 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.892634 9837 catalog_manager.cc:5484] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:46.893573 10132 consensus_queue.cc:260] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.899711 10211 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.900130 10211 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.901181 10132 ts_tablet_manager.cc:1398] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.050s user 0.005s sys 0.000s
I20240426 09:54:46.902153 10132 tablet_bootstrap.cc:492] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:46.902554 10211 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:46.904686 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:46.905445 10013 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:46.898579 10131 tablet_bootstrap.cc:654] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.904809 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:46.906642 9938 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:46.907660 10055 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:46.908552 10211 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:46.908926 10211 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:46.909852 10211 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.913661 10309 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.914139 10309 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.916095 10309 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.916707 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:46.917450 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:54:46.918042 9982 leader_election.cc:341] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
W20240426 09:54:46.918623 9979 leader_election.cc:341] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.919065 9979 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
W20240426 09:54:46.919723 9904 consensus_peers.cc:485] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 -> Peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Couldn't send request to peer d3a480e831d649139ff016fb68aee3ee. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:46.918901 10303 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.920580 10309 raft_consensus.cc:2726] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:46.922461 10130 tablet_bootstrap.cc:492] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:46.922147 10193 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:46.923252 10130 ts_tablet_manager.cc:1367] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.071s user 0.032s sys 0.008s
I20240426 09:54:46.926141 10130 raft_consensus.cc:348] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.926813 10130 raft_consensus.cc:374] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.927079 10130 raft_consensus.cc:729] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.927793 10130 consensus_queue.cc:260] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.922833 10303 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.930176 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
W20240426 09:54:46.931401 9904 leader_election.cc:341] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.931871 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:46.932636 10015 raft_consensus.cc:2379] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:54:46.932976 10184 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.933420 10184 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.937494 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:46.938118 9938 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:46.938823 10130 ts_tablet_manager.cc:1398] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20240426 09:54:46.939756 10130 tablet_bootstrap.cc:492] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:46.941553 10303 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.918417 10211 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.939800 9982 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:46.945596 10132 tablet_bootstrap.cc:654] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.946087 10331 raft_consensus.cc:2781] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:46.946449 10331 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:46.946758 10331 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.947880 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:46.948406 9938 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.949184 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:46.949838 10015 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.953641 10130 tablet_bootstrap.cc:654] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:46.956820 9938 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:46.957334 10331 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.957698 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:46.957653 10211 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:46.959189 10184 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.959231 10056 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:46.960338 10211 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:46.960286 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:46.960994 10211 raft_consensus.cc:686] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:46.961088 10089 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:46.922046 10016 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:46.960232 9904 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:46.963837 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:46.964351 9938 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.961895 10211 consensus_queue.cc:237] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.965590 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:46.966109 10089 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.964185 9906 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:46.961812 10331 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:46.970732 10139 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:46.971387 9938 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:46.971616 10193 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:46.972098 10193 raft_consensus.cc:686] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:54:46.972223 9982 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:46.972790 10331 raft_consensus.cc:2781] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:46.973129 10331 raft_consensus.cc:686] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:46.973693 10331 consensus_queue.cc:237] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.972885 10193 consensus_queue.cc:237] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.976204 10132 tablet_bootstrap.cc:492] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:46.976670 10132 ts_tablet_manager.cc:1367] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.075s user 0.012s sys 0.005s
I20240426 09:54:46.979354 10139 raft_consensus.cc:2726] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:46.980567 10139 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:46.979200 10132 raft_consensus.cc:348] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.981133 10132 raft_consensus.cc:374] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.980964 10139 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.981463 10132 raft_consensus.cc:729] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.982932 10132 consensus_queue.cc:260] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.990934 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:46.991616 10131 tablet_bootstrap.cc:492] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:46.991540 10012 raft_consensus.cc:2454] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:46.992045 10131 ts_tablet_manager.cc:1367] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.119s user 0.008s sys 0.000s
I20240426 09:54:46.993940 10131 raft_consensus.cc:348] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:46.994450 10131 raft_consensus.cc:374] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:46.996855 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:46.997191 9906 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:54:46.998250 9904 leader_election.cc:341] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:46.998858 10132 ts_tablet_manager.cc:1398] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20240426 09:54:46.994678 10131 raft_consensus.cc:729] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:46.999830 10132 tablet_bootstrap.cc:492] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:46.958930 10015 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:47.001468 10131 consensus_queue.cc:260] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.000362 10303 raft_consensus.cc:2781] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:46.999264 10139 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:47.007505 9837 catalog_manager.cc:5484] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:47.032886 10292 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:47.033385 10292 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.026301 10303 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:47.048645 10292 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:47.056617 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:47.057698 10131 ts_tablet_manager.cc:1398] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.065s user 0.016s sys 0.010s
I20240426 09:54:47.058341 10131 tablet_bootstrap.cc:492] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:47.060283 10130 tablet_bootstrap.cc:492] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:47.060600 10130 ts_tablet_manager.cc:1367] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.121s user 0.018s sys 0.010s
I20240426 09:54:47.045337 10303 raft_consensus.cc:3037] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:47.064481 10130 raft_consensus.cc:348] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.069890 10130 raft_consensus.cc:374] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.070170 10130 raft_consensus.cc:729] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.070616 10130 consensus_queue.cc:260] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.067261 10184 consensus_queue.cc:1035] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240426 09:54:47.074084 10130 ts_tablet_manager.cc:1398] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.013s user 0.003s sys 0.003s
I20240426 09:54:47.074798 10130 tablet_bootstrap.cc:492] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:47.062467 10089 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:47.076802 9837 catalog_manager.cc:5484] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:47.081272 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:47.121726 10132 tablet_bootstrap.cc:654] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.125023 10131 tablet_bootstrap.cc:654] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.130177 9838 catalog_manager.cc:5484] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:47.142623 10301 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:47.143201 10301 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.145604 10301 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:47.129107 10055 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:47.149274 10313 raft_consensus.cc:2726] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:47.153544 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:47.155452 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:54:47.160718 9906 leader_election.cc:341] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:47.166388 10331 consensus_queue.cc:1035] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240426 09:54:47.172863 9904 leader_election.cc:341] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:47.173301 9904 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:47.162621 10303 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.177839 10297 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:47.179453 10297 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.180060 10193 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:47.179796 10313 consensus_queue.cc:1035] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:47.180773 10193 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.174367 10301 raft_consensus.cc:2726] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:47.182971 10132 tablet_bootstrap.cc:492] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:47.183421 10132 ts_tablet_manager.cc:1367] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.184s user 0.018s sys 0.001s
I20240426 09:54:47.182354 10297 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:47.183817 10193 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:47.184767 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:47.190155 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:47.185971 10132 raft_consensus.cc:348] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.191593 10132 raft_consensus.cc:374] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.191891 10132 raft_consensus.cc:729] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.191203 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:47.192361 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:47.192616 10132 consensus_queue.cc:260] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.192458 10013 raft_consensus.cc:3037] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
W20240426 09:54:47.193544 10056 leader_election.cc:341] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:47.194139 10130 tablet_bootstrap.cc:654] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.192988 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:47.194972 10132 ts_tablet_manager.cc:1398] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20240426 09:54:47.196036 10132 tablet_bootstrap.cc:492] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:47.198729 10015 raft_consensus.cc:2379] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:54:47.203612 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:47.199163 10303 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:54:47.207003 9904 leader_election.cc:341] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:47.208010 9906 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:47.209935 10303 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:47.214372 10013 raft_consensus.cc:2454] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:47.215762 10303 raft_consensus.cc:2726] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:47.241176 10131 tablet_bootstrap.cc:492] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:47.284160 10132 tablet_bootstrap.cc:654] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.268242 9906 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:47.290797 10303 raft_consensus.cc:2781] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:47.290736 10297 consensus_queue.cc:1035] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 09:54:47.288864 10131 ts_tablet_manager.cc:1367] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.231s user 0.011s sys 0.007s
I20240426 09:54:47.291396 10303 raft_consensus.cc:686] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:54:47.293340 10303 consensus_queue.cc:237] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.294209 10297 consensus_queue.cc:1035] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240426 09:54:47.255570 10055 leader_election.cc:341] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:47.295178 10055 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:47.295253 10131 raft_consensus.cc:348] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.295940 10297 raft_consensus.cc:2726] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:47.296085 10131 raft_consensus.cc:374] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.296640 10131 raft_consensus.cc:729] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.297235 10131 consensus_queue.cc:260] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.300144 10131 ts_tablet_manager.cc:1398] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20240426 09:54:47.303231 10130 tablet_bootstrap.cc:492] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:47.305491 10130 ts_tablet_manager.cc:1367] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.231s user 0.009s sys 0.003s
I20240426 09:54:47.304916 10131 tablet_bootstrap.cc:492] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:47.311622 10297 consensus_queue.cc:1035] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:47.316267 10130 raft_consensus.cc:348] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.317026 10130 raft_consensus.cc:374] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.317312 10130 raft_consensus.cc:729] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.318289 10130 consensus_queue.cc:260] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.320328 10130 ts_tablet_manager.cc:1398] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20240426 09:54:47.321158 10130 tablet_bootstrap.cc:492] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:47.339344 10297 consensus_queue.cc:1035] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:47.343350 10131 tablet_bootstrap.cc:654] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.350464 9838 catalog_manager.cc:5484] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:47.361732 10137 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:47.362361 10137 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.368314 10088 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:47.367645 10137 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:47.369922 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:54:47.371495 9979 leader_election.cc:341] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:47.378607 10193 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:47.379649 10089 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:47.384189 10137 consensus_queue.cc:1035] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240426 09:54:47.389468 10193 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:47.395464 10297 consensus_queue.cc:1035] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:47.396652 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:47.399586 10132 tablet_bootstrap.cc:492] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:47.400038 10132 ts_tablet_manager.cc:1367] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.204s user 0.053s sys 0.045s
W20240426 09:54:47.402184 9982 leader_election.cc:341] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:47.402622 9982 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:47.402333 10132 raft_consensus.cc:348] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.404798 10132 raft_consensus.cc:374] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.405068 10132 raft_consensus.cc:729] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.403622 10137 raft_consensus.cc:2726] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:47.412547 10132 consensus_queue.cc:260] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.422266 10132 ts_tablet_manager.cc:1398] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20240426 09:54:47.423147 10132 tablet_bootstrap.cc:492] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:47.423429 10130 tablet_bootstrap.cc:654] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.450620 10331 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:47.451217 10331 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.458536 10331 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:47.467069 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:54:47.468281 9979 leader_election.cc:341] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:47.470464 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:47.484488 10131 tablet_bootstrap.cc:492] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
W20240426 09:54:47.484925 9982 leader_election.cc:341] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:47.485527 9982 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:47.486485 10331 raft_consensus.cc:2726] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:47.487294 10131 ts_tablet_manager.cc:1367] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.183s user 0.005s sys 0.023s
I20240426 09:54:47.492878 10131 raft_consensus.cc:348] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.493350 10131 raft_consensus.cc:374] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.493536 10131 raft_consensus.cc:729] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.493970 10131 consensus_queue.cc:260] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.500912 10137 consensus_queue.cc:1035] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:47.511586 10131 ts_tablet_manager.cc:1398] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.024s user 0.003s sys 0.002s
I20240426 09:54:47.512270 10131 tablet_bootstrap.cc:492] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:47.512343 10130 tablet_bootstrap.cc:492] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:47.512745 10130 ts_tablet_manager.cc:1367] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.192s user 0.007s sys 0.002s
I20240426 09:54:47.514312 10130 raft_consensus.cc:348] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.514786 10130 raft_consensus.cc:374] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.514997 10130 raft_consensus.cc:729] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.515478 10130 consensus_queue.cc:260] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.516595 10132 tablet_bootstrap.cc:654] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.516947 10130 ts_tablet_manager.cc:1398] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240426 09:54:47.517004 10131 tablet_bootstrap.cc:654] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.517786 10130 tablet_bootstrap.cc:492] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:47.523466 10130 tablet_bootstrap.cc:654] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.537921 10132 tablet_bootstrap.cc:492] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:47.538345 10132 ts_tablet_manager.cc:1367] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.115s user 0.009s sys 0.009s
I20240426 09:54:47.541155 10132 raft_consensus.cc:348] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.541852 10132 raft_consensus.cc:374] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.542191 10132 raft_consensus.cc:729] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.542834 10132 consensus_queue.cc:260] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.545091 10132 ts_tablet_manager.cc:1398] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240426 09:54:47.546043 10132 tablet_bootstrap.cc:492] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:47.551626 10132 tablet_bootstrap.cc:654] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.552387 10131 tablet_bootstrap.cc:492] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:47.552843 10131 ts_tablet_manager.cc:1367] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.009s
I20240426 09:54:47.555198 10131 raft_consensus.cc:348] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.555871 10131 raft_consensus.cc:374] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.556119 10131 raft_consensus.cc:729] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.558009 10292 consensus_queue.cc:1035] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:47.556658 10131 consensus_queue.cc:260] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.563087 10313 consensus_queue.cc:1035] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:47.565351 10132 tablet_bootstrap.cc:492] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:47.565820 10132 ts_tablet_manager.cc:1367] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.003s
I20240426 09:54:47.568607 10132 raft_consensus.cc:348] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.569243 10132 raft_consensus.cc:374] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.569494 10132 raft_consensus.cc:729] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.572170 10132 consensus_queue.cc:260] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.575402 10131 ts_tablet_manager.cc:1398] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20240426 09:54:47.577083 10131 tablet_bootstrap.cc:492] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:47.577247 10132 ts_tablet_manager.cc:1398] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20240426 09:54:47.578114 10132 tablet_bootstrap.cc:492] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:47.583555 10132 tablet_bootstrap.cc:654] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.583675 10131 tablet_bootstrap.cc:654] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.584928 10130 tablet_bootstrap.cc:492] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:47.585458 10130 ts_tablet_manager.cc:1367] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.068s user 0.024s sys 0.016s
I20240426 09:54:47.587894 10130 raft_consensus.cc:348] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.624989 10130 raft_consensus.cc:374] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.625609 10130 raft_consensus.cc:729] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.627138 10130 consensus_queue.cc:260] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.631489 10132 tablet_bootstrap.cc:492] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:47.632527 10132 ts_tablet_manager.cc:1367] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.003s
I20240426 09:54:47.632663 10130 ts_tablet_manager.cc:1398] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.047s user 0.009s sys 0.000s
I20240426 09:54:47.637065 10132 raft_consensus.cc:348] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.637984 10132 raft_consensus.cc:374] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.638314 10132 raft_consensus.cc:729] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.639321 10132 consensus_queue.cc:260] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:47.641615 9887 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 09:54:47.642812 10130 tablet_bootstrap.cc:492] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:47.642827 10132 ts_tablet_manager.cc:1398] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
I20240426 09:54:47.647259 10132 tablet_bootstrap.cc:492] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:47.650372 10130 tablet_bootstrap.cc:654] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.653317 10132 tablet_bootstrap.cc:654] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.660059 10137 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:47.660687 10137 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.662946 10137 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:47.664371 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:47.664841 10300 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:47.665468 10300 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.665494 10088 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:47.668164 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:47.668802 10300 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:47.668951 9941 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:47.670351 9979 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:47.672605 10131 tablet_bootstrap.cc:492] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:47.672958 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:47.673797 10015 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:47.673332 10131 ts_tablet_manager.cc:1367] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.096s user 0.007s sys 0.005s
I20240426 09:54:47.679158 10331 raft_consensus.cc:2781] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:47.679551 10331 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:47.679859 10331 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:47.680557 10130 tablet_bootstrap.cc:492] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:47.681069 10130 ts_tablet_manager.cc:1367] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.039s user 0.011s sys 0.008s
I20240426 09:54:47.683671 10131 raft_consensus.cc:348] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.684510 10131 raft_consensus.cc:374] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.684690 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:47.685292 10088 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:47.686028 10130 raft_consensus.cc:348] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.686431 9904 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:47.686722 10130 raft_consensus.cc:374] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.687043 10130 raft_consensus.cc:729] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.687165 10300 raft_consensus.cc:2781] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:47.687530 10300 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:47.687863 10300 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:47.687657 10130 consensus_queue.cc:260] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.690574 10130 ts_tablet_manager.cc:1398] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240426 09:54:47.690968 10131 raft_consensus.cc:729] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.691601 10131 consensus_queue.cc:260] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.695603 10131 ts_tablet_manager.cc:1398] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.019s user 0.005s sys 0.003s
I20240426 09:54:47.696874 10131 tablet_bootstrap.cc:492] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:47.697628 10300 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.700103 10300 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:47.702700 10131 tablet_bootstrap.cc:654] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.703648 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:47.704216 10088 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:47.711122 10300 consensus_queue.cc:1035] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:47.711841 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:47.712687 10088 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:47.713819 9904 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:47.719223 10193 raft_consensus.cc:2781] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:47.719717 10193 raft_consensus.cc:686] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:54:47.720733 10193 consensus_queue.cc:237] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.719717 10331 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.726874 10331 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:47.728760 10015 raft_consensus.cc:2379] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:54:47.732149 10130 tablet_bootstrap.cc:492] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:47.736194 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:47.737493 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:47.747810 10085 raft_consensus.cc:2379] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 1: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:54:47.748423 10131 tablet_bootstrap.cc:492] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:47.748874 10131 ts_tablet_manager.cc:1367] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.052s user 0.018s sys 0.013s
I20240426 09:54:47.753561 10131 raft_consensus.cc:348] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.754097 10131 raft_consensus.cc:374] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.754338 10131 raft_consensus.cc:729] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.754916 10131 consensus_queue.cc:260] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.755784 9979 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:47.765389 10130 tablet_bootstrap.cc:654] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.765460 10131 ts_tablet_manager.cc:1398] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20240426 09:54:47.762809 10137 raft_consensus.cc:2726] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240426 09:54:47.769717 10131 tablet_bootstrap.cc:492] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:47.780429 10300 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:47.783982 10132 tablet_bootstrap.cc:492] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:47.787195 10132 ts_tablet_manager.cc:1367] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.140s user 0.027s sys 0.019s
I20240426 09:54:47.787104 10300 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.787799 10297 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:47.786338 10131 tablet_bootstrap.cc:654] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.788771 10297 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.791350 10297 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:47.791893 10300 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:47.792706 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:47.794551 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:47.797654 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:47.798650 9941 raft_consensus.cc:2454] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:47.794930 10132 raft_consensus.cc:348] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.800287 10132 raft_consensus.cc:374] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.800582 10132 raft_consensus.cc:729] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.801491 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:47.802816 10085 raft_consensus.cc:2454] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:47.803856 10085 raft_consensus.cc:3037] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:47.802409 10132 consensus_queue.cc:260] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.805567 9904 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:54:47.795428 10015 raft_consensus.cc:2454] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:47.807914 10132 ts_tablet_manager.cc:1398] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20240426 09:54:47.808929 10132 tablet_bootstrap.cc:492] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:47.809405 10055 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:47.810194 10297 raft_consensus.cc:2781] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:54:47.810598 10297 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:47.811014 10297 raft_consensus.cc:3037] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:47.818943 10139 consensus_queue.cc:1035] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 09:54:47.820535 10130 tablet_bootstrap.cc:492] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:47.820930 10130 ts_tablet_manager.cc:1367] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.089s user 0.007s sys 0.007s
I20240426 09:54:47.821924 10303 raft_consensus.cc:2781] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:54:47.822288 10303 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:47.822628 10303 raft_consensus.cc:3037] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:47.829648 10313 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:47.830214 10313 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.843757 10331 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:47.844336 10331 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.854841 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:47.855494 9940 raft_consensus.cc:2454] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:47.865346 9982 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:47.866955 10309 raft_consensus.cc:2781] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:47.867316 10309 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:47.867606 10309 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:47.873883 10131 tablet_bootstrap.cc:492] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:47.874325 10131 ts_tablet_manager.cc:1367] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.105s user 0.008s sys 0.021s
I20240426 09:54:47.874514 10184 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:47.875056 10184 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.877092 10184 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:47.876957 10131 raft_consensus.cc:348] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.877673 10131 raft_consensus.cc:374] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.877934 10131 raft_consensus.cc:729] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.878610 10131 consensus_queue.cc:260] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.879371 10132 tablet_bootstrap.cc:654] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.880069 10331 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:47.885486 10131 ts_tablet_manager.cc:1398] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.011s user 0.002s sys 0.004s
I20240426 09:54:47.886624 10131 tablet_bootstrap.cc:492] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:47.886862 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:47.887730 10130 raft_consensus.cc:348] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.888401 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:47.889086 10085 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:47.890545 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:47.891306 9940 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:47.892789 10131 tablet_bootstrap.cc:654] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.892719 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:47.893234 10313 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:47.888397 10130 raft_consensus.cc:374] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.896075 10303 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:47.897389 10055 leader_election.cc:341] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:47.897982 10130 raft_consensus.cc:729] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.901890 10303 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:47.903402 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:47.902508 9982 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:47.904052 9941 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:47.904794 10331 raft_consensus.cc:2781] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:47.905181 10331 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:47.905517 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:47.905506 10331 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:47.906841 10131 tablet_bootstrap.cc:492] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:47.907274 10131 ts_tablet_manager.cc:1367] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.003s
I20240426 09:54:47.909524 10056 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:54:47.909587 10131 raft_consensus.cc:348] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.910332 10131 raft_consensus.cc:374] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.910620 10131 raft_consensus.cc:729] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.911104 10292 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:47.911451 10292 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:47.911599 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:47.911911 10131 consensus_queue.cc:260] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.912381 10292 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:47.912799 10016 raft_consensus.cc:3037] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:47.913534 10131 ts_tablet_manager.cc:1398] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 09:54:47.914374 10131 tablet_bootstrap.cc:492] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:47.918112 10301 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:47.918948 10301 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.919510 10131 tablet_bootstrap.cc:654] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
W20240426 09:54:47.920608 9979 leader_election.cc:341] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:47.898157 9838 catalog_manager.cc:5484] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:47.922549 10292 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.930614 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:47.931190 9940 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:47.931510 10131 tablet_bootstrap.cc:492] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:47.931918 10131 ts_tablet_manager.cc:1367] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20240426 09:54:47.934665 10131 raft_consensus.cc:348] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.935405 10131 raft_consensus.cc:374] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:47.935693 10131 raft_consensus.cc:729] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:47.936328 10131 consensus_queue.cc:260] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.937824 10301 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:47.938347 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:47.939285 10085 raft_consensus.cc:2379] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:54:47.940124 10131 ts_tablet_manager.cc:1398] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240426 09:54:47.941051 10131 tablet_bootstrap.cc:492] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:47.946707 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:47.947533 10015 raft_consensus.cc:2379] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:54:47.948632 10131 tablet_bootstrap.cc:654] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:47.949223 10130 consensus_queue.cc:260] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.949594 9906 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:47.951470 10301 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:47.952744 10016 raft_consensus.cc:2454] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:54:47.953794 9906 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:47.954631 10139 raft_consensus.cc:2781] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:54:47.955089 10139 raft_consensus.cc:686] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:54:47.955456 10211 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:47.956323 10130 ts_tablet_manager.cc:1398] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.135s user 0.009s sys 0.000s
I20240426 09:54:47.955911 10211 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.955938 10139 consensus_queue.cc:237] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.957551 10130 tablet_bootstrap.cc:492] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:47.957984 10211 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:47.959324 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:47.959419 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:47.960028 10016 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:47.960085 9939 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:47.961061 10055 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:47.961766 10211 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:54:47.962092 10211 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:47.962409 10211 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:47.965042 9940 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:47.978096 10211 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.983292 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:47.983868 10016 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:47.986519 10309 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.988472 10309 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:47.989828 10016 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:54:47.993446 10211 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.017457 10055 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:48.019951 10211 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:54:48.020408 10211 raft_consensus.cc:686] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 2 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:48.021260 10211 consensus_queue.cc:237] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.025635 10331 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.994365 10292 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:47.995047 10313 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.035600 10313 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:47.997692 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:48.039171 10313 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.007623 10301 raft_consensus.cc:2726] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:48.008392 10297 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.012573 10184 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.040829 10184 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.043440 10090 raft_consensus.cc:2379] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 2: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:54:48.046093 10331 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:48.046422 10297 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.056351 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:48.060052 10015 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:48.079866 10130 tablet_bootstrap.cc:654] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
W20240426 09:54:48.080226 10055 leader_election.cc:341] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.080927 10055 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:48.082731 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:48.083629 10015 raft_consensus.cc:2379] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:54:48.088788 10292 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.089239 10292 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.091023 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:48.091814 9940 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:48.096226 10438 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:48.096598 10438 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:48.096691 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:48.096947 10438 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.096983 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:48.097173 10309 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.097664 10088 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.099583 9940 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:54:48.097685 10309 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.103433 10132 tablet_bootstrap.cc:492] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:48.103881 10132 ts_tablet_manager.cc:1367] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.295s user 0.037s sys 0.028s
I20240426 09:54:48.103557 10438 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.104704 10309 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:48.105540 10438 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.106462 10088 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:48.091899 10184 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:48.108989 10131 tablet_bootstrap.cc:492] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:48.109444 10131 ts_tablet_manager.cc:1367] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.169s user 0.085s sys 0.045s
I20240426 09:54:48.111763 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:48.111845 10131 raft_consensus.cc:348] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.112455 10016 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:48.112707 10131 raft_consensus.cc:374] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.113121 10131 raft_consensus.cc:729] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.113193 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:48.113655 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:48.113821 9939 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
W20240426 09:54:48.113229 9979 leader_election.cc:341] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.114188 10016 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.114646 9979 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:48.114583 10131 consensus_queue.cc:260] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.116379 10131 ts_tablet_manager.cc:1398] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240426 09:54:48.117007 10184 raft_consensus.cc:2781] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:48.118103 10184 raft_consensus.cc:686] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:48.118247 10131 tablet_bootstrap.cc:492] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:48.119702 10184 consensus_queue.cc:237] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.120919 9939 raft_consensus.cc:2454] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:48.121891 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.124017 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.124763 10131 tablet_bootstrap.cc:654] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.126585 10016 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:48.126734 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.106376 10132 raft_consensus.cc:348] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.168298 10132 raft_consensus.cc:374] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.168773 10132 raft_consensus.cc:729] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.149156 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:48.170070 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:48.148092 9982 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:48.167107 10130 tablet_bootstrap.cc:492] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:48.170794 10090 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
W20240426 09:54:48.171347 9982 leader_election.cc:341] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.144851 10056 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:54:48.151125 10301 consensus_queue.cc:1035] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20240426 09:54:48.172529 10309 raft_consensus.cc:2781] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:48.172173 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.174566 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:48.175613 10130 ts_tablet_manager.cc:1367] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.218s user 0.012s sys 0.000s
I20240426 09:54:48.143507 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:48.091091 10292 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.178433 10130 raft_consensus.cc:348] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.179323 10130 raft_consensus.cc:374] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.179589 10130 raft_consensus.cc:729] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.180272 10130 consensus_queue.cc:260] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.180999 10055 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:48.181790 10437 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:48.182307 10437 raft_consensus.cc:686] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:48.137219 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:48.183399 10438 raft_consensus.cc:2781] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:48.183145 10437 consensus_queue.cc:237] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.183470 10055 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:48.183836 10438 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:48.184273 10438 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.184535 9979 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
W20240426 09:54:48.184871 10056 leader_election.cc:341] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
W20240426 09:54:48.185431 9979 leader_election.cc:341] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.185834 9979 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:48.184286 10292 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:48.189463 10292 raft_consensus.cc:686] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:48.187335 10132 consensus_queue.cc:260] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.191715 10292 consensus_queue.cc:237] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.188442 10331 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:48.194945 10331 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:48.195277 10331 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.183131 9940 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.170874 9941 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:48.208042 10445 raft_consensus.cc:2726] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:48.210615 10309 raft_consensus.cc:686] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:48.211598 10309 consensus_queue.cc:237] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.217984 10211 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.218475 10211 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.219668 10056 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:48.220577 10430 raft_consensus.cc:2726] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240426 09:54:48.220790 10056 leader_election.cc:341] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.220896 10211 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.196134 10438 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:48.221781 10056 leader_election.cc:341] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.222635 10438 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.233170 9940 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:48.236224 10130 ts_tablet_manager.cc:1398] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.060s user 0.005s sys 0.001s
I20240426 09:54:48.237303 10130 tablet_bootstrap.cc:492] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:48.238615 10132 ts_tablet_manager.cc:1398] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.134s user 0.010s sys 0.000s
I20240426 09:54:48.239611 10132 tablet_bootstrap.cc:492] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:48.241480 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.244149 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:48.244679 10016 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.245154 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:48.245750 10015 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:48.246065 10131 tablet_bootstrap.cc:492] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:48.246501 10131 ts_tablet_manager.cc:1367] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.128s user 0.052s sys 0.031s
I20240426 09:54:48.247256 10055 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:48.249123 10131 raft_consensus.cc:348] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.249832 10131 raft_consensus.cc:374] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.250120 10131 raft_consensus.cc:729] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.250804 10131 consensus_queue.cc:260] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.251368 10016 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:48.251781 10438 raft_consensus.cc:2781] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:48.252182 10438 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:48.252543 10438 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.253050 10055 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:48.253486 10132 tablet_bootstrap.cc:654] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.253921 10437 raft_consensus.cc:2781] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:48.253038 10131 ts_tablet_manager.cc:1398] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240426 09:54:48.255328 10131 tablet_bootstrap.cc:492] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:48.256070 10437 raft_consensus.cc:686] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:48.256978 10437 consensus_queue.cc:237] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.259094 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:48.259655 10132 tablet_bootstrap.cc:492] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:48.261003 10132 ts_tablet_manager.cc:1367] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.000s
I20240426 09:54:48.261636 10313 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.262060 10313 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:48.262609 10056 leader_election.cc:341] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.263355 10132 raft_consensus.cc:348] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.263931 10313 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.264034 10132 raft_consensus.cc:374] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.264415 10132 raft_consensus.cc:729] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.264899 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:48.265120 10132 consensus_queue.cc:260] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.267154 10132 ts_tablet_manager.cc:1398] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240426 09:54:48.268101 10132 tablet_bootstrap.cc:492] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:48.269714 10131 tablet_bootstrap.cc:654] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
W20240426 09:54:48.271296 10056 leader_election.cc:341] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.274197 10132 tablet_bootstrap.cc:654] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.275591 10331 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.277740 10131 tablet_bootstrap.cc:492] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:48.278209 10131 ts_tablet_manager.cc:1367] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.007s
I20240426 09:54:48.278802 10132 tablet_bootstrap.cc:492] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:48.279206 10132 ts_tablet_manager.cc:1367] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240426 09:54:48.280974 10131 raft_consensus.cc:348] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.281729 10131 raft_consensus.cc:374] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.282042 10131 raft_consensus.cc:729] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.281905 10132 raft_consensus.cc:348] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.282951 10132 raft_consensus.cc:374] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.282706 10131 consensus_queue.cc:260] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.283257 10132 raft_consensus.cc:729] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.283967 10132 consensus_queue.cc:260] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.285071 10131 ts_tablet_manager.cc:1398] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240426 09:54:48.286024 10131 tablet_bootstrap.cc:492] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:48.286139 10132 ts_tablet_manager.cc:1398] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240426 09:54:48.286933 10132 tablet_bootstrap.cc:492] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:48.291301 10131 tablet_bootstrap.cc:654] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.291800 10132 tablet_bootstrap.cc:654] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.293670 10438 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.295699 10438 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.296566 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:48.297154 10015 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.297763 10131 tablet_bootstrap.cc:492] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:48.298115 10131 ts_tablet_manager.cc:1367] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240426 09:54:48.298377 10130 tablet_bootstrap.cc:654] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.299927 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.300266 10131 raft_consensus.cc:348] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.300849 9940 raft_consensus.cc:2379] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 1: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:54:48.300932 10131 raft_consensus.cc:374] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.301333 10131 raft_consensus.cc:729] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.301889 10131 consensus_queue.cc:260] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.303354 10331 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:48.303889 10131 ts_tablet_manager.cc:1398] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240426 09:54:48.304836 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:48.304832 10131 tablet_bootstrap.cc:492] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:48.306286 9837 catalog_manager.cc:5484] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:48.309751 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:48.310286 9940 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.311981 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:48.312531 10088 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.313349 10015 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:48.315981 10056 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:48.317042 10438 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
W20240426 09:54:48.317366 10056 leader_election.cc:341] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.320116 10055 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:48.320803 10292 raft_consensus.cc:2781] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:48.321269 10292 raft_consensus.cc:686] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:48.322017 10292 consensus_queue.cc:237] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.325414 10132 tablet_bootstrap.cc:492] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:48.325851 10132 ts_tablet_manager.cc:1367] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.000s
I20240426 09:54:48.327881 10438 raft_consensus.cc:2726] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:48.329303 10132 raft_consensus.cc:348] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.330057 10132 raft_consensus.cc:374] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.330322 10132 raft_consensus.cc:729] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.315428 10131 tablet_bootstrap.cc:654] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.330911 10132 consensus_queue.cc:260] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.333024 10132 ts_tablet_manager.cc:1398] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240426 09:54:48.333564 10193 consensus_queue.cc:1035] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:48.333940 10132 tablet_bootstrap.cc:492] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:48.337218 10088 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:48.338508 9979 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:48.339318 10445 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:48.339848 10132 tablet_bootstrap.cc:654] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.339810 10445 raft_consensus.cc:686] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:48.343847 10445 consensus_queue.cc:237] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.330700 9839 catalog_manager.cc:5484] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 0 to 2, leader changed from <none> to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 2 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:48.355465 10438 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.362360 10438 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.316006 9940 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:48.366657 10437 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.367100 10437 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.368994 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:48.369616 10015 raft_consensus.cc:2454] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:48.370846 10132 tablet_bootstrap.cc:492] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:48.371052 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:48.371510 10132 ts_tablet_manager.cc:1367] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.006s
I20240426 09:54:48.371695 10015 raft_consensus.cc:2454] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:48.368999 10437 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.370745 10055 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:48.372793 10437 raft_consensus.cc:2781] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:48.373184 10437 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:48.373494 10437 raft_consensus.cc:3037] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.373553 10055 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:48.374218 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.374378 10292 raft_consensus.cc:2781] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:48.374343 10132 raft_consensus.cc:348] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.374902 10292 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:48.375171 10132 raft_consensus.cc:374] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.375232 10292 raft_consensus.cc:3037] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
W20240426 09:54:48.375356 10056 leader_election.cc:341] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.380414 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.380995 9940 raft_consensus.cc:2454] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:48.383618 10292 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.368114 10438 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.393529 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:48.394762 10015 raft_consensus.cc:3037] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.402354 10292 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.402665 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:54:48.405592 10056 leader_election.cc:341] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.406004 10015 raft_consensus.cc:2454] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:48.406580 9838 catalog_manager.cc:5484] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:48.407102 10055 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:48.408687 10130 tablet_bootstrap.cc:492] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:48.409127 10130 ts_tablet_manager.cc:1367] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.172s user 0.013s sys 0.006s
I20240426 09:54:48.409309 10438 raft_consensus.cc:2781] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:48.375488 10132 raft_consensus.cc:729] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.413211 10438 raft_consensus.cc:686] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:48.413337 10132 consensus_queue.cc:260] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.415105 10438 consensus_queue.cc:237] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.410409 10313 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.419075 10313 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.416185 10132 ts_tablet_manager.cc:1398] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.044s user 0.006s sys 0.000s
I20240426 09:54:48.420763 10313 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.421229 10131 tablet_bootstrap.cc:492] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:48.422080 10131 ts_tablet_manager.cc:1367] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.117s user 0.033s sys 0.036s
I20240426 09:54:48.422150 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.422329 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:54:48.423316 10056 leader_election.cc:341] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
W20240426 09:54:48.423743 10055 leader_election.cc:341] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.424122 10055 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:48.424382 10131 raft_consensus.cc:348] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.425067 10131 raft_consensus.cc:374] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.411713 10130 raft_consensus.cc:348] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.411783 9838 catalog_manager.cc:5484] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:48.425348 10313 raft_consensus.cc:2726] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:48.410615 10437 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.427270 10130 raft_consensus.cc:374] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.427559 10130 raft_consensus.cc:729] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.428141 10130 consensus_queue.cc:260] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.430162 10130 ts_tablet_manager.cc:1398] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s
I20240426 09:54:48.430989 10130 tablet_bootstrap.cc:492] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:48.421329 10132 tablet_bootstrap.cc:492] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:48.425323 10131 raft_consensus.cc:729] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.436044 10131 consensus_queue.cc:260] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.437027 9837 catalog_manager.cc:5484] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:48.434906 10445 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.440810 10445 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.441970 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:48.442502 9940 raft_consensus.cc:3037] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.442700 10445 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:48.443704 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:48.445114 10131 ts_tablet_manager.cc:1398] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.023s user 0.000s sys 0.004s
W20240426 09:54:48.445041 9979 leader_election.cc:341] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.445592 9837 catalog_manager.cc:5484] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:48.447639 9837 catalog_manager.cc:5484] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 2, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 2 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:48.449396 10331 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.449473 9837 catalog_manager.cc:5484] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:48.449914 10331 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.451525 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.451890 10331 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:48.437116 10437 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.449504 10131 tablet_bootstrap.cc:492] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
W20240426 09:54:48.453819 9982 leader_election.cc:341] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.454236 9982 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:48.455304 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:48.456141 10450 raft_consensus.cc:2726] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:48.456944 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:54:48.457918 9979 leader_election.cc:341] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.448941 9940 raft_consensus.cc:2454] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:48.459522 10056 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:48.463199 10437 raft_consensus.cc:2781] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:48.463660 10437 raft_consensus.cc:686] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
W20240426 09:54:48.466778 9982 leader_election.cc:341] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.467322 9982 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:48.468478 10331 raft_consensus.cc:2726] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:48.469475 10130 tablet_bootstrap.cc:654] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.474412 10132 tablet_bootstrap.cc:654] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.470806 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:48.475745 10015 raft_consensus.cc:3037] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.474788 10437 consensus_queue.cc:237] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.483856 10090 raft_consensus.cc:1261] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240426 09:54:48.488806 10193 consensus_queue.cc:1035] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 09:54:48.490061 10131 tablet_bootstrap.cc:654] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.503154 10445 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.504119 10445 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.509536 10445 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:48.510248 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.510473 10015 raft_consensus.cc:2454] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:48.517202 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:54:48.515931 9982 leader_election.cc:341] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.520311 10130 tablet_bootstrap.cc:492] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:48.520792 10130 ts_tablet_manager.cc:1367] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.090s user 0.012s sys 0.009s
I20240426 09:54:48.525058 9940 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20240426 09:54:48.527611 9979 leader_election.cc:341] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.526670 10292 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:48.528059 9979 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:48.529012 10445 raft_consensus.cc:2726] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:48.527613 10130 raft_consensus.cc:348] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.530205 10130 raft_consensus.cc:374] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.530356 10131 tablet_bootstrap.cc:492] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:48.530464 10130 raft_consensus.cc:729] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.530927 10131 ts_tablet_manager.cc:1367] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.082s user 0.010s sys 0.008s
I20240426 09:54:48.531682 10130 consensus_queue.cc:260] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.533277 10130 ts_tablet_manager.cc:1398] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20240426 09:54:48.533198 10131 raft_consensus.cc:348] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.533893 10131 raft_consensus.cc:374] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.534176 10131 raft_consensus.cc:729] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.535248 10131 consensus_queue.cc:260] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.536643 10130 tablet_bootstrap.cc:492] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:48.537369 10131 ts_tablet_manager.cc:1398] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20240426 09:54:48.538247 10131 tablet_bootstrap.cc:492] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:48.539005 10438 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.539438 10438 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.541332 10438 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.543282 10130 tablet_bootstrap.cc:654] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.543342 10014 raft_consensus.cc:1261] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240426 09:54:48.544307 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:48.544790 10139 consensus_queue.cc:1035] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240426 09:54:48.545432 10055 leader_election.cc:341] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.547571 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.550351 10131 tablet_bootstrap.cc:654] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
W20240426 09:54:48.550910 10056 leader_election.cc:341] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.551308 10130 tablet_bootstrap.cc:492] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:48.551294 10056 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:48.551834 10130 ts_tablet_manager.cc:1367] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.001s
I20240426 09:54:48.552328 10438 raft_consensus.cc:2726] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:48.557664 10132 tablet_bootstrap.cc:492] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:48.558106 10132 ts_tablet_manager.cc:1367] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.137s user 0.009s sys 0.011s
I20240426 09:54:48.560312 10132 raft_consensus.cc:348] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.560943 10132 raft_consensus.cc:374] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.561247 10132 raft_consensus.cc:729] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.561842 10132 consensus_queue.cc:260] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.564092 10132 ts_tablet_manager.cc:1398] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 09:54:48.564953 10132 tablet_bootstrap.cc:492] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:48.565161 10193 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.565652 10193 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.567606 10193 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:48.569233 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:48.569737 10132 tablet_bootstrap.cc:654] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.569977 10014 raft_consensus.cc:2379] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:54:48.570839 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:48.570151 10130 raft_consensus.cc:348] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.572199 10130 raft_consensus.cc:374] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.572456 10130 raft_consensus.cc:729] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.573197 10130 consensus_queue.cc:260] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.574637 10014 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240426 09:54:48.575465 10130 ts_tablet_manager.cc:1398] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.023s user 0.005s sys 0.005s
I20240426 09:54:48.576428 10130 tablet_bootstrap.cc:492] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:48.576218 10438 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:48.573822 9904 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:48.582388 10139 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.582584 10130 tablet_bootstrap.cc:654] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.588011 9837 catalog_manager.cc:5484] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:48.591051 10131 tablet_bootstrap.cc:492] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:48.591492 10131 ts_tablet_manager.cc:1367] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.003s
I20240426 09:54:48.589929 9837 catalog_manager.cc:5484] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:48.596081 10131 raft_consensus.cc:348] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.596750 10131 raft_consensus.cc:374] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.597030 10131 raft_consensus.cc:729] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.597616 10131 consensus_queue.cc:260] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.599617 10131 ts_tablet_manager.cc:1398] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240426 09:54:48.600503 10131 tablet_bootstrap.cc:492] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:48.602449 10139 raft_consensus.cc:2726] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:48.605729 10131 tablet_bootstrap.cc:654] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.606212 10445 consensus_queue.cc:1035] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:48.612920 9837 catalog_manager.cc:5484] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:48.614040 10445 consensus_queue.cc:1035] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:48.615269 10450 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.618359 10450 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.620591 10450 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:48.620877 10131 tablet_bootstrap.cc:492] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:48.621323 10131 ts_tablet_manager.cc:1367] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.007s
I20240426 09:54:48.623756 10131 raft_consensus.cc:348] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.624236 10292 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.624424 10131 raft_consensus.cc:374] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.624704 10131 raft_consensus.cc:729] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.625362 10131 consensus_queue.cc:260] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.625824 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:48.624717 10292 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:48.627468 9979 leader_election.cc:341] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.628646 10292 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.629774 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.630023 10131 ts_tablet_manager.cc:1398] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240426 09:54:48.632133 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:48.633100 10131 tablet_bootstrap.cc:492] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
W20240426 09:54:48.633376 10055 leader_election.cc:341] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
W20240426 09:54:48.634215 9982 leader_election.cc:341] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.634697 9982 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:48.635684 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.636260 10331 raft_consensus.cc:2726] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240426 09:54:48.637125 10056 leader_election.cc:341] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.637518 10056 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:48.638384 10438 raft_consensus.cc:2726] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:48.641294 10131 tablet_bootstrap.cc:654] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.658543 10331 consensus_queue.cc:1035] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240426 09:54:48.682374 10056 consensus_peers.cc:485] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee -> Peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Couldn't send request to peer f12521b6d51242629cf4297864df00b9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:48.689110 10438 consensus_queue.cc:1035] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:48.698880 10132 tablet_bootstrap.cc:492] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:48.699559 10132 ts_tablet_manager.cc:1367] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.135s user 0.009s sys 0.001s
I20240426 09:54:48.703004 10438 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.703466 10438 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.703913 10130 tablet_bootstrap.cc:492] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:48.704352 10130 ts_tablet_manager.cc:1367] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.128s user 0.039s sys 0.060s
I20240426 09:54:48.705303 10438 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.707456 10130 raft_consensus.cc:348] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.708057 10438 consensus_queue.cc:1035] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:48.708772 10130 raft_consensus.cc:374] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.709070 10130 raft_consensus.cc:729] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.709651 10130 consensus_queue.cc:260] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.710706 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:48.711673 10130 ts_tablet_manager.cc:1398] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.007s user 0.001s sys 0.006s
I20240426 09:54:48.712098 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.712576 10130 tablet_bootstrap.cc:492] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:48.713004 10446 consensus_queue.cc:1035] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240426 09:54:48.715166 10055 leader_election.cc:341] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.715775 10292 consensus_queue.cc:1035] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:48.719087 10130 tablet_bootstrap.cc:654] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
W20240426 09:54:48.722443 10056 leader_election.cc:341] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.722863 10056 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:48.723466 10131 tablet_bootstrap.cc:492] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:48.723887 10131 ts_tablet_manager.cc:1367] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.091s user 0.031s sys 0.033s
I20240426 09:54:48.723793 10292 raft_consensus.cc:2726] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:48.726212 10131 raft_consensus.cc:348] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.726874 10131 raft_consensus.cc:374] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.727129 10131 raft_consensus.cc:729] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.727681 10131 consensus_queue.cc:260] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.729092 10131 ts_tablet_manager.cc:1398] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240426 09:54:48.729946 10131 tablet_bootstrap.cc:492] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
W20240426 09:54:48.729892 10055 consensus_peers.cc:485] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee -> Peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Couldn't send request to peer 826f03b2fb9c4b9483623b041a1bf80e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:48.754216 10445 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.754740 10445 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.761596 10131 tablet_bootstrap.cc:654] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.762802 10445 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:48.767699 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:48.768389 10090 raft_consensus.cc:2454] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:48.770298 9937 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.771920 10131 tablet_bootstrap.cc:492] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:48.772194 9979 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:48.772342 10131 ts_tablet_manager.cc:1367] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.005s
I20240426 09:54:48.773164 10445 raft_consensus.cc:2781] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:48.773593 10445 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:48.773998 10445 raft_consensus.cc:3037] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.774825 10131 raft_consensus.cc:348] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.775481 10131 raft_consensus.cc:374] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.775733 10131 raft_consensus.cc:729] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.776273 10131 consensus_queue.cc:260] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:48.777748 9887 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 09:54:48.778707 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.780823 10438 consensus_queue.cc:1035] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:48.781179 10131 ts_tablet_manager.cc:1398] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20240426 09:54:48.782532 10131 tablet_bootstrap.cc:492] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:48.787178 10132 raft_consensus.cc:348] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:48.787685 9982 leader_election.cc:341] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.787971 10131 tablet_bootstrap.cc:654] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.787905 10132 raft_consensus.cc:374] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.796795 10132 raft_consensus.cc:729] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.806221 10132 consensus_queue.cc:260] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.810487 10131 tablet_bootstrap.cc:492] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:48.845912 10446 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.846088 10131 ts_tablet_manager.cc:1367] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.064s user 0.008s sys 0.008s
I20240426 09:54:48.846406 10446 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.848670 10131 raft_consensus.cc:348] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.849309 10131 raft_consensus.cc:374] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.849568 10131 raft_consensus.cc:729] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.850204 10446 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:48.810696 10445 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.811843 10292 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:48.852905 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.853533 10445 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:48.850112 10131 consensus_queue.cc:260] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.855654 10131 ts_tablet_manager.cc:1398] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.009s user 0.000s sys 0.004s
I20240426 09:54:48.856592 10131 tablet_bootstrap.cc:492] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:48.857432 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:54:48.858434 9982 leader_election.cc:341] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
W20240426 09:54:48.861606 9982 leader_election.cc:341] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.862160 10132 ts_tablet_manager.cc:1398] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.162s user 0.005s sys 0.000s
I20240426 09:54:48.862468 9938 raft_consensus.cc:3037] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.863174 10132 tablet_bootstrap.cc:492] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:48.874380 10131 tablet_bootstrap.cc:654] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.875502 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:48.876143 10090 raft_consensus.cc:3037] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.883759 10313 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:48.885680 10090 raft_consensus.cc:2454] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:48.882501 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:48.886907 10085 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:48.891407 9979 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:48.893075 9979 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:48.893602 10313 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.894033 10313 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.896999 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:48.897578 10016 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:48.896163 10313 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.899041 10055 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:48.900005 10313 raft_consensus.cc:2781] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:48.900838 10437 consensus_queue.cc:1035] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:48.902891 10313 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:48.903364 10313 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.907636 10450 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:48.907677 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.908288 10450 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:48.907941 10445 raft_consensus.cc:2781] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:48.908672 10450 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.909068 10445 raft_consensus.cc:686] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:48.909832 10445 consensus_queue.cc:237] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.910434 10193 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.910969 10193 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.913678 10193 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:48.914856 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:54:48.915441 10056 leader_election.cc:341] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.916242 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:48.918105 9904 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:48.918794 10193 raft_consensus.cc:2726] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:54:48.920096 10132 tablet_bootstrap.cc:654] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:48.925352 10437 consensus_queue.cc:1035] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:48.926318 10303 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.926738 10303 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.928495 10303 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:48.934012 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:48.935266 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:48.936818 9904 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:48.937013 10313 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.937785 10193 raft_consensus.cc:2726] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:48.933276 10450 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.941112 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:54:48.943886 9982 leader_election.cc:341] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.946653 10331 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:48.949862 10445 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.952337 10445 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.949934 10438 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:48.953620 10438 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.955328 10438 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.955999 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.956256 10445 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:48.957284 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:48.958370 9939 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:48.959316 10131 tablet_bootstrap.cc:492] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:48.959210 10450 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:48.959971 10131 ts_tablet_manager.cc:1367] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.104s user 0.018s sys 0.004s
I20240426 09:54:48.961041 9982 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:54:48.962898 10056 leader_election.cc:341] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:48.964044 10446 raft_consensus.cc:2781] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:54:48.964417 10446 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:48.964728 10446 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:48.966732 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:48.967324 10016 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:48.966804 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:48.972093 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:48.972553 10088 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.972637 10085 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:48.962481 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:48.980845 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:48.981365 10016 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:48.986438 10016 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:48.990413 10184 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:48.994053 10130 tablet_bootstrap.cc:492] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:48.994493 10130 ts_tablet_manager.cc:1367] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.282s user 0.095s sys 0.119s
I20240426 09:54:48.996902 10130 raft_consensus.cc:348] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.997606 10130 raft_consensus.cc:374] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:48.997892 10130 raft_consensus.cc:729] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:48.998559 10130 consensus_queue.cc:260] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:48.971110 10313 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:48.999899 10055 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:49.000639 10438 raft_consensus.cc:2781] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:49.000990 10438 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:49.001323 10438 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.001227 10055 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:49.001809 10130 ts_tablet_manager.cc:1398] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240426 09:54:49.001951 10437 raft_consensus.cc:2781] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:49.002468 10437 raft_consensus.cc:686] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:49.003240 10437 consensus_queue.cc:237] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.004438 10088 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:49.007098 9979 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:49.010430 9838 catalog_manager.cc:5484] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:49.012836 10331 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:49.013379 10331 raft_consensus.cc:686] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:49.014183 10331 consensus_queue.cc:237] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:49.016342 10056 leader_election.cc:341] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:49.024363 10130 tablet_bootstrap.cc:492] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:48.996876 10446 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.036077 10313 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:49.036546 10313 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.038300 10313 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:49.041266 10438 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.043576 10131 raft_consensus.cc:348] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.044085 10131 raft_consensus.cc:374] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:49.044255 10131 raft_consensus.cc:729] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:49.046106 10131 consensus_queue.cc:260] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.047374 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:49.047889 9940 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:49.049355 10131 ts_tablet_manager.cc:1398] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.089s user 0.004s sys 0.000s
I20240426 09:54:49.050196 10131 tablet_bootstrap.cc:492] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee: Bootstrap starting.
I20240426 09:54:49.059715 9940 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:54:49.061427 9982 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:49.070930 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:49.072783 10446 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:49.071540 10014 raft_consensus.cc:2454] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:49.074018 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:49.074533 10130 tablet_bootstrap.cc:654] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:49.074605 10090 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:54:49.065641 10450 raft_consensus.cc:2781] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:54:49.084612 10450 raft_consensus.cc:686] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:49.085494 10450 consensus_queue.cc:237] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.078636 10438 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:49.088321 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:49.088817 10016 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.094810 10055 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:49.099226 10090 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:54:49.096148 9839 catalog_manager.cc:5484] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:49.099625 10016 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:49.102402 10438 raft_consensus.cc:2781] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:49.103340 10438 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:49.103735 10438 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.102820 10055 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:49.105406 10437 raft_consensus.cc:2781] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:49.105826 10437 raft_consensus.cc:686] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:49.106547 10437 consensus_queue.cc:237] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.143709 9839 catalog_manager.cc:5484] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 2, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 2 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:49.147691 10440 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:49.159955 10440 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.161767 10440 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:49.117529 9837 catalog_manager.cc:5484] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:49.102507 10499 consensus_queue.cc:1035] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.057s
I20240426 09:54:49.128594 10292 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:49.163995 10292 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.168059 10438 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.170212 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:49.171716 10131 tablet_bootstrap.cc:654] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee: Neither blocks nor log segments found. Creating new log.
W20240426 09:54:49.172199 10055 leader_election.cc:341] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:49.174491 10132 tablet_bootstrap.cc:492] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:49.182485 10132 ts_tablet_manager.cc:1367] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.319s user 0.024s sys 0.023s
I20240426 09:54:49.183257 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:49.185561 10132 raft_consensus.cc:348] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.186220 10132 raft_consensus.cc:374] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:49.186475 10132 raft_consensus.cc:729] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:49.186677 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:49.188148 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:49.175032 10438 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:49.190528 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:49.176075 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:49.178315 10292 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:49.179239 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:49.180989 10501 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:49.191485 10132 consensus_queue.cc:260] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.192013 10501 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.193049 10130 tablet_bootstrap.cc:492] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:49.193454 10130 ts_tablet_manager.cc:1367] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.169s user 0.021s sys 0.010s
I20240426 09:54:49.193866 10501 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:49.191548 10015 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.195052 9839 catalog_manager.cc:5484] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:49.197906 10132 ts_tablet_manager.cc:1398] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20240426 09:54:49.198192 10130 raft_consensus.cc:348] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.198864 10130 raft_consensus.cc:374] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:49.199128 10130 raft_consensus.cc:729] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:49.199077 10132 tablet_bootstrap.cc:492] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:49.200155 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:54:49.201550 10055 leader_election.cc:341] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:49.201817 10015 raft_consensus.cc:2454] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:49.202729 10055 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
W20240426 09:54:49.203486 10056 leader_election.cc:341] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:49.204066 10440 raft_consensus.cc:2781] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:49.204550 10440 raft_consensus.cc:686] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
W20240426 09:54:49.205627 10056 leader_election.cc:341] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:49.205790 10132 tablet_bootstrap.cc:654] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:49.209465 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:49.208777 10056 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:49.210983 10499 raft_consensus.cc:2726] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:49.199734 10130 consensus_queue.cc:260] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:49.212416 10056 leader_election.cc:341] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:49.213840 10130 ts_tablet_manager.cc:1398] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20240426 09:54:49.214658 10130 tablet_bootstrap.cc:492] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
W20240426 09:54:49.215150 10056 leader_election.cc:341] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:49.209090 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:49.216682 10014 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:49.205271 10440 consensus_queue.cc:237] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:49.215737 10056 leader_election.cc:341] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:49.203490 10501 consensus_queue.cc:1035] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20240426 09:54:49.218611 10055 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:49.219969 10130 tablet_bootstrap.cc:654] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:49.220108 10292 raft_consensus.cc:2781] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:49.222533 10292 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:49.222937 10292 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.233574 9940 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.242231 10132 tablet_bootstrap.cc:492] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:49.242719 10132 ts_tablet_manager.cc:1367] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.013s
I20240426 09:54:49.245198 10132 raft_consensus.cc:348] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:49.245666 10056 leader_election.cc:341] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:49.246524 10056 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:54:49.249402 10438 raft_consensus.cc:2726] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:49.245879 10132 raft_consensus.cc:374] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:49.251014 10132 raft_consensus.cc:729] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:49.251636 10132 consensus_queue.cc:260] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.253698 10130 tablet_bootstrap.cc:492] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:49.254150 10130 ts_tablet_manager.cc:1367] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.008s
I20240426 09:54:49.257771 10132 ts_tablet_manager.cc:1398] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.015s user 0.004s sys 0.001s
I20240426 09:54:49.258718 9837 catalog_manager.cc:5484] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:49.260444 10132 tablet_bootstrap.cc:492] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:49.267756 10132 tablet_bootstrap.cc:654] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:49.287731 10130 raft_consensus.cc:348] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.289844 10130 raft_consensus.cc:374] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:49.290112 10130 raft_consensus.cc:729] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:49.290807 10130 consensus_queue.cc:260] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.293294 10130 ts_tablet_manager.cc:1398] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.039s user 0.005s sys 0.000s
I20240426 09:54:49.294164 10130 tablet_bootstrap.cc:492] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:49.230253 10292 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.298898 10130 tablet_bootstrap.cc:654] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:49.301286 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:49.301882 10015 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.315881 10015 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:49.318759 10055 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:49.319943 10506 raft_consensus.cc:2781] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:49.320408 10506 raft_consensus.cc:686] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:49.321252 10506 consensus_queue.cc:237] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.340582 10303 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:49.341126 10303 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.344638 10303 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:49.345512 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:49.352702 10292 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:49.354681 9940 raft_consensus.cc:3037] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.359439 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:49.367091 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:49.367858 10090 raft_consensus.cc:2379] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
W20240426 09:54:49.370561 10056 leader_election.cc:341] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:49.376133 9837 catalog_manager.cc:5484] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:49.379902 10131 tablet_bootstrap.cc:492] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee: No bootstrap required, opened a new log
I20240426 09:54:49.380506 10131 ts_tablet_manager.cc:1367] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee: Time spent bootstrapping tablet: real 0.331s user 0.017s sys 0.040s
I20240426 09:54:49.380008 10132 tablet_bootstrap.cc:492] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:49.381848 10132 ts_tablet_manager.cc:1367] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.122s user 0.051s sys 0.048s
I20240426 09:54:49.383807 10131 raft_consensus.cc:348] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.384117 10132 raft_consensus.cc:348] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.385202 10132 raft_consensus.cc:374] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:49.385526 10132 raft_consensus.cc:729] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:49.384500 9904 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:49.384953 10131 raft_consensus.cc:374] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:49.386545 10131 raft_consensus.cc:729] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Initialized, Role: FOLLOWER
I20240426 09:54:49.386343 10132 consensus_queue.cc:260] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.387238 10131 consensus_queue.cc:260] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.387794 10130 tablet_bootstrap.cc:492] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:49.388238 10130 ts_tablet_manager.cc:1367] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.094s user 0.022s sys 0.036s
I20240426 09:54:49.389451 10131 ts_tablet_manager.cc:1398] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240426 09:54:49.391988 10130 raft_consensus.cc:348] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.392685 10130 raft_consensus.cc:374] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:49.392938 10130 raft_consensus.cc:729] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:49.392923 10331 consensus_queue.cc:1035] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:49.393654 10130 consensus_queue.cc:260] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.395845 10303 raft_consensus.cc:2726] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:49.397902 10193 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:49.398128 10130 ts_tablet_manager.cc:1398] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240426 09:54:49.398299 10193 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.399037 10130 tablet_bootstrap.cc:492] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:49.401386 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:54:49.403435 9906 leader_election.cc:341] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:49.404178 10193 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:49.407935 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:49.408519 10088 raft_consensus.cc:2454] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
I20240426 09:54:49.409193 10132 ts_tablet_manager.cc:1398] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20240426 09:54:49.409781 10130 tablet_bootstrap.cc:654] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:49.411027 9904 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:54:49.411936 10193 raft_consensus.cc:2781] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:49.412243 10193 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:49.410710 10132 tablet_bootstrap.cc:492] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:49.412530 10193 raft_consensus.cc:3037] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.418844 10193 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.419095 10309 consensus_queue.cc:1035] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:49.420419 10130 tablet_bootstrap.cc:492] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:49.420851 10130 ts_tablet_manager.cc:1367] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.002s
I20240426 09:54:49.421398 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:49.421821 10088 raft_consensus.cc:3037] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.423718 10193 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:49.423621 10130 raft_consensus.cc:348] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.424302 10130 raft_consensus.cc:374] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:49.424563 10130 raft_consensus.cc:729] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:49.425091 10130 consensus_queue.cc:260] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.428875 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:49.429584 10130 ts_tablet_manager.cc:1398] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20240426 09:54:49.430155 9906 leader_election.cc:341] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:49.434022 10088 raft_consensus.cc:2454] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:49.435546 10450 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:49.436000 10450 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.437070 9904 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:54:49.439216 10130 tablet_bootstrap.cc:492] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:49.441637 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:49.442445 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:49.447044 9940 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:49.447407 10085 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:49.448069 9982 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:49.447904 10303 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:49.449318 10309 raft_consensus.cc:2781] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:49.450222 10309 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:49.437711 10450 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:49.450625 10309 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.451438 9940 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.448920 10303 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.452893 10130 tablet_bootstrap.cc:654] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:49.453424 10303 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:49.461194 10450 consensus_queue.cc:1035] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:49.461923 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:49.462602 10016 raft_consensus.cc:2379] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:54:49.437758 10139 raft_consensus.cc:2781] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:49.463510 10139 raft_consensus.cc:686] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:54:49.464294 10139 consensus_queue.cc:237] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.465456 10309 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.467031 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:49.468210 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:49.474184 10085 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.474948 10184 consensus_queue.cc:1035] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:49.475540 10132 tablet_bootstrap.cc:654] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:49.480055 10085 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:49.481972 10132 tablet_bootstrap.cc:492] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:49.482389 10132 ts_tablet_manager.cc:1367] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.001s
I20240426 09:54:49.484023 9979 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:49.485438 10445 raft_consensus.cc:2781] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:49.485847 10445 raft_consensus.cc:686] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:49.486492 10445 consensus_queue.cc:237] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.489006 10132 raft_consensus.cc:348] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.489637 10132 raft_consensus.cc:374] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:49.491545 10132 raft_consensus.cc:729] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:49.492236 10132 consensus_queue.cc:260] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.493918 10132 ts_tablet_manager.cc:1398] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240426 09:54:49.494596 10132 tablet_bootstrap.cc:492] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
I20240426 09:54:49.495599 10450 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:49.496026 10450 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.468812 10309 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:49.470455 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:49.472683 10130 tablet_bootstrap.cc:492] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:49.499545 9941 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.499655 10130 ts_tablet_manager.cc:1367] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.061s user 0.019s sys 0.004s
I20240426 09:54:49.499883 10309 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:49.501945 10130 raft_consensus.cc:348] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.502581 10130 raft_consensus.cc:374] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:49.502899 10130 raft_consensus.cc:729] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:49.503422 10450 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:49.500301 10309 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.504949 9837 catalog_manager.cc:5484] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:49.506536 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:49.507042 9941 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:49.499118 10132 tablet_bootstrap.cc:654] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:49.506482 10130 consensus_queue.cc:260] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.510092 9904 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:49.514313 10130 ts_tablet_manager.cc:1398] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240426 09:54:49.514348 10506 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:49.515619 10506 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.515154 10130 tablet_bootstrap.cc:492] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:49.518054 10506 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:49.518849 10193 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.519315 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:49.519908 10085 raft_consensus.cc:1261] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240426 09:54:49.519908 10014 raft_consensus.cc:2454] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 0.
I20240426 09:54:49.521807 10055 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:49.524583 10193 raft_consensus.cc:2726] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:49.526016 10506 raft_consensus.cc:2781] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:49.526135 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:49.526389 10506 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:49.526741 10506 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.526758 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:49.530892 9839 catalog_manager.cc:5484] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:49.531785 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:49.521162 10309 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:49.535187 9982 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:49.536504 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:54:49.537930 9982 leader_election.cc:341] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
W20240426 09:54:49.543107 10056 leader_election.cc:341] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:49.543030 10184 raft_consensus.cc:2726] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:49.525213 10450 consensus_queue.cc:1035] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20240426 09:54:49.547622 10506 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.548713 10088 raft_consensus.cc:2379] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 1: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:54:49.551785 10506 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:49.553050 10130 tablet_bootstrap.cc:654] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:49.553540 9979 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:49.556010 10445 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.559345 10132 tablet_bootstrap.cc:492] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:49.687366 10132 ts_tablet_manager.cc:1367] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.193s user 0.012s sys 0.000s
I20240426 09:54:49.559911 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:49.685855 10437 consensus_queue.cc:1035] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240426 09:54:49.567818 10445 raft_consensus.cc:2726] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:49.567924 10506 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.143s
I20240426 09:54:49.695981 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:49.695739 10450 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:49.720145 10016 raft_consensus.cc:2454] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:54:49.696118 10499 consensus_queue.cc:1035] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
I20240426 09:54:49.719111 10450 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.732721 10450 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:49.735109 10132 raft_consensus.cc:348] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.735970 10132 raft_consensus.cc:374] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:49.736269 10132 raft_consensus.cc:729] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:49.736910 10132 consensus_queue.cc:260] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.749166 10132 ts_tablet_manager.cc:1398] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.061s user 0.007s sys 0.000s
W20240426 09:54:49.754065 10525 log.cc:924] Time spent T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.217s user 0.000s sys 0.001s
I20240426 09:54:49.762970 10130 tablet_bootstrap.cc:492] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:49.765110 10130 ts_tablet_manager.cc:1367] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.250s user 0.018s sys 0.034s
I20240426 09:54:49.760739 10132 tablet_bootstrap.cc:492] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e: Bootstrap starting.
W20240426 09:54:49.764005 10056 leader_election.cc:341] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Illegal state: must be running to vote when last-logged opid is not known
I20240426 09:54:49.802199 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:49.802999 10084 raft_consensus.cc:2454] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:49.831465 10130 raft_consensus.cc:348] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.834823 10130 raft_consensus.cc:374] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:49.835125 10130 raft_consensus.cc:729] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:49.835839 10130 consensus_queue.cc:260] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.857100 10130 ts_tablet_manager.cc:1398] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.092s user 0.007s sys 0.011s
I20240426 09:54:49.857522 10533 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:49.857992 10533 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.862811 9935 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.868592 10132 tablet_bootstrap.cc:654] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:49.872440 10533 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:49.870147 10438 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240426 09:54:49.890527 10139 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:49.890959 10139 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.892622 10139 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:49.893963 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:49.895444 10013 raft_consensus.cc:2454] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
W20240426 09:54:49.902107 10056 consensus_peers.cc:485] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee -> Peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Couldn't send request to peer f12521b6d51242629cf4297864df00b9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:49.904119 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:49.906978 9906 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:49.908957 10139 raft_consensus.cc:2781] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:49.909457 10139 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:49.909771 10139 raft_consensus.cc:3037] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.910852 9937 raft_consensus.cc:1261] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240426 09:54:49.909198 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:49.920692 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:49.920856 10089 raft_consensus.cc:2454] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 0.
W20240426 09:54:49.925558 10056 consensus_peers.cc:485] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee -> Peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Couldn't send request to peer f12521b6d51242629cf4297864df00b9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:49.933827 10499 consensus_queue.cc:1035] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240426 09:54:49.936475 10532 consensus_queue.cc:1035] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:49.944078 9979 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:49.945470 10446 raft_consensus.cc:2781] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:49.945819 10446 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:49.946147 10446 raft_consensus.cc:3037] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.948204 10055 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:54:49.956516 10139 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.958320 10139 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:49.958925 10437 raft_consensus.cc:2781] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:49.959877 10446 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.961699 10437 raft_consensus.cc:686] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:54:49.962832 10130 tablet_bootstrap.cc:492] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:49.962648 10437 consensus_queue.cc:237] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:49.959795 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:49.973459 10130 tablet_bootstrap.cc:654] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:49.933873 10450 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.040s
I20240426 09:54:49.977473 9936 raft_consensus.cc:2379] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 1: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:54:49.974344 10085 raft_consensus.cc:3037] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.974161 10438 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240426 09:54:49.992142 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:49.990437 10014 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:49.997771 10087 raft_consensus.cc:2379] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 1: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:54:50.000207 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:49.997840 10446 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:49.985798 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:54:50.010475 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:50.013530 10013 raft_consensus.cc:2379] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 1: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:54:50.023876 10532 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:50.026510 10132 tablet_bootstrap.cc:492] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e: No bootstrap required, opened a new log
I20240426 09:54:50.029356 10085 raft_consensus.cc:2454] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:54:50.035713 10132 ts_tablet_manager.cc:1367] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent bootstrapping tablet: real 0.275s user 0.020s sys 0.007s
W20240426 09:54:50.041348 9887 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 09:54:50.042640 9904 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:54:50.053992 9979 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:50.055707 10529 raft_consensus.cc:2781] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:50.056180 10529 raft_consensus.cc:686] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:54:50.065779 10184 raft_consensus.cc:3037] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:50.071267 10139 consensus_queue.cc:1035] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:50.071936 9979 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:54:50.085327 9839 catalog_manager.cc:5484] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:54:50.087755 10132 raft_consensus.cc:348] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:50.088454 10132 raft_consensus.cc:374] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:50.088764 10132 raft_consensus.cc:729] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Initialized, Role: FOLLOWER
I20240426 09:54:50.092399 10132 consensus_queue.cc:260] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:54:50.094964 10539 log.cc:924] Time spent T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20240426 09:54:50.095283 10529 consensus_queue.cc:237] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:50.079186 10533 raft_consensus.cc:2726] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240426 09:54:50.105587 10132 ts_tablet_manager.cc:1398] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent starting tablet: real 0.070s user 0.005s sys 0.001s
I20240426 09:54:50.113919 10184 raft_consensus.cc:2726] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:50.121198 10130 tablet_bootstrap.cc:492] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:50.121656 10130 ts_tablet_manager.cc:1367] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.159s user 0.016s sys 0.008s
I20240426 09:54:50.132405 10139 consensus_queue.cc:1035] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:50.155010 10130 raft_consensus.cc:348] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:50.155745 10130 raft_consensus.cc:374] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:50.155956 10130 raft_consensus.cc:729] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:50.157948 10130 consensus_queue.cc:260] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:50.195031 10130 ts_tablet_manager.cc:1398] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.073s user 0.013s sys 0.001s
I20240426 09:54:50.196025 10130 tablet_bootstrap.cc:492] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:50.219966 10130 tablet_bootstrap.cc:654] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:50.223398 9836 catalog_manager.cc:5484] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:50.231318 10130 tablet_bootstrap.cc:492] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:50.231694 10130 ts_tablet_manager.cc:1367] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.008s
I20240426 09:54:50.233862 10130 raft_consensus.cc:348] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:50.234463 10130 raft_consensus.cc:374] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:50.234707 10130 raft_consensus.cc:729] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:50.235289 10130 consensus_queue.cc:260] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:50.237143 10130 ts_tablet_manager.cc:1398] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20240426 09:54:50.238019 10130 tablet_bootstrap.cc:492] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:50.243124 10130 tablet_bootstrap.cc:654] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:50.248732 10130 tablet_bootstrap.cc:492] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:50.249013 10130 ts_tablet_manager.cc:1367] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240426 09:54:50.251296 10130 raft_consensus.cc:348] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:50.251961 10130 raft_consensus.cc:374] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:50.252213 10130 raft_consensus.cc:729] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:50.252700 10130 consensus_queue.cc:260] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:50.254559 10130 ts_tablet_manager.cc:1398] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20240426 09:54:50.255789 10130 tablet_bootstrap.cc:492] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9: Bootstrap starting.
I20240426 09:54:50.262743 10130 tablet_bootstrap.cc:654] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9: Neither blocks nor log segments found. Creating new log.
I20240426 09:54:50.285245 10130 tablet_bootstrap.cc:492] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9: No bootstrap required, opened a new log
I20240426 09:54:50.285598 10130 ts_tablet_manager.cc:1367] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.004s
I20240426 09:54:50.287200 10130 raft_consensus.cc:348] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:50.287662 10130 raft_consensus.cc:374] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:54:50.287845 10130 raft_consensus.cc:729] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Initialized, Role: FOLLOWER
I20240426 09:54:50.288288 10130 consensus_queue.cc:260] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:50.289651 10130 ts_tablet_manager.cc:1398] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240426 09:54:50.334549 10538 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:50.334960 10538 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:50.336522 10538 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:50.337230 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:50.337476 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:50.338838 9904 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:50.339354 10538 raft_consensus.cc:2726] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:50.355001 9938 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:50.397676 10538 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:50.398537 9938 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:50.398129 10538 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:50.400545 10538 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:50.401084 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:54:50.401333 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:50.402623 9904 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:54:50.403259 10538 raft_consensus.cc:2726] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 09:54:50.404841 10499 consensus_queue.cc:1035] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:50.414746 10292 consensus_queue.cc:1035] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:50.434441 10184 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:54:50.434880 10184 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:50.436513 10184 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:50.437510 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:54:50.437624 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:54:50.438099 10089 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:50.438239 9936 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 0.
I20240426 09:54:50.438899 9979 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:54:50.439589 10184 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:54:50.439901 10184 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:54:50.440251 10184 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:50.445806 10184 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:50.447795 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:54:50.447985 10184 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:50.448302 9936 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:50.448308 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:54:50.449010 10086 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:50.455080 9936 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:50.455940 10086 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:54:50.455968 9982 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:54:50.456825 10184 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:54:50.457479 10184 raft_consensus.cc:686] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:54:50.458150 10184 consensus_queue.cc:237] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:54:50.472795 9839 catalog_manager.cc:5484] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:50.604009 10193 consensus_queue.cc:1035] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:50.614965 10193 consensus_queue.cc:1035] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 09:54:50.644470 9936 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:54:50.939137 10184 consensus_queue.cc:1035] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:54:50.953294 10184 consensus_queue.cc:1035] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 09:54:52.127072 8758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:54:52.189958 10574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:54:52.190945 10577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:54:52.191260 10575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:54:52.192158 8758 server_base.cc:1034] running on GCE node
I20240426 09:54:52.194065 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 09:54:52.194373 8758 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 09:54:52.194619 8758 hybrid_clock.cc:648] HybridClock initialized: now 1714125292194599 us; error 0 us; skew 500 ppm
I20240426 09:54:52.195426 8758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:54:52.201627 8758 webserver.cc:460] Webserver started at http://127.8.141.132:38503/ using document root <none> and password file <none>
I20240426 09:54:52.202097 8758 fs_manager.cc:359] Metadata directory not provided
I20240426 09:54:52.202282 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:54:52.202543 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:54:52.203671 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-3-root/instance:
uuid: "1fee3da01ebc463396397057fdb81a68"
format_stamp: "Formatted at 2024-04-26 09:54:52 on dist-test-slave-5txv"
I20240426 09:54:52.207841 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240426 09:54:52.211236 10582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:52.211992 8758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240426 09:54:52.212255 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-3-root
uuid: "1fee3da01ebc463396397057fdb81a68"
format_stamp: "Formatted at 2024-04-26 09:54:52 on dist-test-slave-5txv"
I20240426 09:54:52.212512 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:54:52.225255 8758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 09:54:52.226688 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:54:52.229588 8758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:54:52.232270 8758 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:54:52.232491 8758 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:52.232745 8758 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:54:52.232899 8758 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 09:54:52.288604 8758 rpc_server.cc:307] RPC server started. Bound to: 127.8.141.132:34313
I20240426 09:54:52.288697 10644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.141.132:34313 every 8 connection(s)
I20240426 09:54:52.292209 8758 auto_leader_rebalancer-test.cc:405] add a tserver: 1fee3da01ebc463396397057fdb81a68
I20240426 09:54:52.302702 10645 heartbeater.cc:347] Connected to a master server at 127.8.141.190:45863
I20240426 09:54:52.303158 10645 heartbeater.cc:464] Registering TS with master...
I20240426 09:54:52.304072 10645 heartbeater.cc:511] Master 127.8.141.190:45863 requested a full tablet report, sending...
I20240426 09:54:52.305708 9836 ts_manager.cc:194] Registered new tserver with Master: 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:54:52.306913 9836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58806
I20240426 09:54:53.259070 10010 consensus_queue.cc:237] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:53.266417 9938 raft_consensus.cc:1261] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:54:53.267675 10086 raft_consensus.cc:1261] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:54:53.268990 10653 consensus_queue.cc:1035] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:53.269832 10649 consensus_queue.cc:1035] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:53.278909 10184 raft_consensus.cc:2932] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
W20240426 09:54:53.281922 9980 consensus_peers.cc:485] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ddc5aa33dc824544ade18161bf2800a6. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:53.284126 9936 raft_consensus.cc:2932] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:53.290881 10089 raft_consensus.cc:2932] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:53.293947 9836 catalog_manager.cc:5484] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:53.305904 10013 consensus_queue.cc:237] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:53.308943 10645 heartbeater.cc:503] Master 127.8.141.190:45863 was elected leader, sending a full tablet report...
W20240426 09:54:53.314410 9980 consensus_peers.cc:485] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c0fe10a2b3746f9909d3596580b6854. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:53.315124 9936 raft_consensus.cc:1261] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:54:53.315224 10089 raft_consensus.cc:1261] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:54:53.318398 10184 consensus_queue.cc:1035] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:53.320463 10184 consensus_queue.cc:1035] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:53.333479 10653 raft_consensus.cc:2932] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:53.334825 9938 raft_consensus.cc:2932] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:53.337088 10089 raft_consensus.cc:2932] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:53.351830 9838 catalog_manager.cc:5484] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:53.370481 10089 consensus_queue.cc:237] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:53.376699 9938 raft_consensus.cc:1261] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:54:53.378305 10013 raft_consensus.cc:1261] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:54:53.379298 10499 consensus_queue.cc:1035] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:53.382436 10499 consensus_queue.cc:1035] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:53.397884 10668 raft_consensus.cc:2932] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
W20240426 09:54:53.403685 10054 consensus_peers.cc:485] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7ef86a50a514e8ba09c76ca674b2100. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:53.403673 9936 raft_consensus.cc:2932] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:53.412021 10013 consensus_queue.cc:237] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:53.416311 9836 catalog_manager.cc:5484] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240426 09:54:53.428438 9980 consensus_peers.cc:485] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b63eef1eb5e4fabb35fb66b0f5a41b4. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:53.430797 10014 raft_consensus.cc:2932] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:53.439982 10089 raft_consensus.cc:1261] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:54:53.442646 10658 consensus_queue.cc:1035] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 09:54:53.445878 9936 raft_consensus.cc:1261] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:54:53.447274 10653 consensus_queue.cc:1035] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:53.450742 10658 raft_consensus.cc:2932] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:53.456902 10089 raft_consensus.cc:2932] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:53.474887 9839 catalog_manager.cc:5484] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:53.475775 9938 raft_consensus.cc:2932] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:53.735674 10677 ts_tablet_manager.cc:902] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:53.754812 10677 tablet_copy_client.cc:308] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.130:44411
I20240426 09:54:53.765801 10026 tablet_copy_service.cc:140] P 826f03b2fb9c4b9483623b041a1bf80e: Received BeginTabletCopySession request for tablet 5c0fe10a2b3746f9909d3596580b6854 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:36762)
I20240426 09:54:53.766356 10026 tablet_copy_service.cc:161] P 826f03b2fb9c4b9483623b041a1bf80e: Beginning new tablet copy session on tablet 5c0fe10a2b3746f9909d3596580b6854 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:36762: session id = 1fee3da01ebc463396397057fdb81a68-5c0fe10a2b3746f9909d3596580b6854
I20240426 09:54:53.774418 10026 tablet_copy_source_session.cc:205] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:54:53.777827 10677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c0fe10a2b3746f9909d3596580b6854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:53.790630 10677 tablet_copy_client.cc:783] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:54:53.791205 10677 tablet_copy_client.cc:647] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:54:53.795944 10677 tablet_copy_client.cc:523] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:54:53.802600 10677 tablet_bootstrap.cc:492] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:54:53.820701 10677 tablet_bootstrap.cc:492] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:54:53.821578 10677 tablet_bootstrap.cc:492] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:54:53.822047 10677 ts_tablet_manager.cc:1367] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.020s user 0.021s sys 0.001s
I20240426 09:54:53.824700 10677 raft_consensus.cc:348] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:53.825325 10677 raft_consensus.cc:729] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:54:53.825874 10677 consensus_queue.cc:260] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:53.828965 10677 ts_tablet_manager.cc:1398] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240426 09:54:53.830350 10026 tablet_copy_service.cc:342] P 826f03b2fb9c4b9483623b041a1bf80e: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-5c0fe10a2b3746f9909d3596580b6854 received from {username='slave'} at 127.0.0.1:36762
I20240426 09:54:53.831120 10026 tablet_copy_service.cc:433] P 826f03b2fb9c4b9483623b041a1bf80e: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-5c0fe10a2b3746f9909d3596580b6854 on tablet 5c0fe10a2b3746f9909d3596580b6854 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:54:53.855796 10677 ts_tablet_manager.cc:902] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:53.858806 10677 tablet_copy_client.cc:308] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.130:44411
I20240426 09:54:53.860518 10026 tablet_copy_service.cc:140] P 826f03b2fb9c4b9483623b041a1bf80e: Received BeginTabletCopySession request for tablet ddc5aa33dc824544ade18161bf2800a6 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:36762)
I20240426 09:54:53.860934 10026 tablet_copy_service.cc:161] P 826f03b2fb9c4b9483623b041a1bf80e: Beginning new tablet copy session on tablet ddc5aa33dc824544ade18161bf2800a6 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:36762: session id = 1fee3da01ebc463396397057fdb81a68-ddc5aa33dc824544ade18161bf2800a6
I20240426 09:54:53.865875 10026 tablet_copy_source_session.cc:205] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:54:53.869366 10677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc5aa33dc824544ade18161bf2800a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:53.880957 10677 tablet_copy_client.cc:783] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:54:53.881439 10677 tablet_copy_client.cc:647] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:54:53.884982 10677 tablet_copy_client.cc:523] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:54:53.893095 10677 tablet_bootstrap.cc:492] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:54:53.915148 10677 tablet_bootstrap.cc:492] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:54:53.915838 10677 tablet_bootstrap.cc:492] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:54:53.916337 10677 ts_tablet_manager.cc:1367] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.009s
I20240426 09:54:53.920317 10677 raft_consensus.cc:348] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:53.921057 10677 raft_consensus.cc:729] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:54:53.921581 10677 consensus_queue.cc:260] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:53.923491 10677 ts_tablet_manager.cc:1398] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20240426 09:54:53.925668 10026 tablet_copy_service.cc:342] P 826f03b2fb9c4b9483623b041a1bf80e: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-ddc5aa33dc824544ade18161bf2800a6 received from {username='slave'} at 127.0.0.1:36762
I20240426 09:54:53.926038 10026 tablet_copy_service.cc:433] P 826f03b2fb9c4b9483623b041a1bf80e: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-ddc5aa33dc824544ade18161bf2800a6 on tablet ddc5aa33dc824544ade18161bf2800a6 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:54:53.954952 10677 ts_tablet_manager.cc:902] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:53.964126 10677 tablet_copy_client.cc:308] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.130:44411
I20240426 09:54:53.965358 10026 tablet_copy_service.cc:140] P 826f03b2fb9c4b9483623b041a1bf80e: Received BeginTabletCopySession request for tablet 7b63eef1eb5e4fabb35fb66b0f5a41b4 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:36762)
I20240426 09:54:53.965749 10026 tablet_copy_service.cc:161] P 826f03b2fb9c4b9483623b041a1bf80e: Beginning new tablet copy session on tablet 7b63eef1eb5e4fabb35fb66b0f5a41b4 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:36762: session id = 1fee3da01ebc463396397057fdb81a68-7b63eef1eb5e4fabb35fb66b0f5a41b4
I20240426 09:54:53.972671 10026 tablet_copy_source_session.cc:205] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:54:53.980089 10677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b63eef1eb5e4fabb35fb66b0f5a41b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:53.999588 10694 ts_tablet_manager.cc:902] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:54.004526 10677 tablet_copy_client.cc:783] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:54:54.005017 10677 tablet_copy_client.cc:647] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:54:54.005636 10694 tablet_copy_client.cc:308] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.131:34821
I20240426 09:54:54.008138 10677 tablet_copy_client.cc:523] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:54:54.018328 10677 tablet_bootstrap.cc:492] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:54:54.031483 10100 tablet_copy_service.cc:140] P d3a480e831d649139ff016fb68aee3ee: Received BeginTabletCopySession request for tablet a7ef86a50a514e8ba09c76ca674b2100 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:48202)
I20240426 09:54:54.032058 10100 tablet_copy_service.cc:161] P d3a480e831d649139ff016fb68aee3ee: Beginning new tablet copy session on tablet a7ef86a50a514e8ba09c76ca674b2100 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:48202: session id = 1fee3da01ebc463396397057fdb81a68-a7ef86a50a514e8ba09c76ca674b2100
I20240426 09:54:54.037470 10677 tablet_bootstrap.cc:492] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:54:54.038213 10677 tablet_bootstrap.cc:492] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:54:54.038800 10677 ts_tablet_manager.cc:1367] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.005s
I20240426 09:54:54.039301 10100 tablet_copy_source_session.cc:205] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:54:54.041078 10677 raft_consensus.cc:348] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:54.041842 10677 raft_consensus.cc:729] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:54:54.042867 10694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ef86a50a514e8ba09c76ca674b2100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:54.042380 10677 consensus_queue.cc:260] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:54.046428 10677 ts_tablet_manager.cc:1398] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240426 09:54:54.047809 10026 tablet_copy_service.cc:342] P 826f03b2fb9c4b9483623b041a1bf80e: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-7b63eef1eb5e4fabb35fb66b0f5a41b4 received from {username='slave'} at 127.0.0.1:36762
I20240426 09:54:54.048195 10026 tablet_copy_service.cc:433] P 826f03b2fb9c4b9483623b041a1bf80e: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-7b63eef1eb5e4fabb35fb66b0f5a41b4 on tablet 7b63eef1eb5e4fabb35fb66b0f5a41b4 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:54:54.063987 10694 tablet_copy_client.cc:783] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:54:54.064498 10694 tablet_copy_client.cc:647] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:54:54.067327 10694 tablet_copy_client.cc:523] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:54:54.073982 10694 tablet_bootstrap.cc:492] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:54:54.090101 10694 tablet_bootstrap.cc:492] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:54:54.090868 10694 tablet_bootstrap.cc:492] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:54:54.091353 10694 ts_tablet_manager.cc:1367] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.004s
I20240426 09:54:54.093721 10694 raft_consensus.cc:348] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:54.094344 10694 raft_consensus.cc:729] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:54:54.094871 10694 consensus_queue.cc:260] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:54.096897 10694 ts_tablet_manager.cc:1398] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20240426 09:54:54.098600 10100 tablet_copy_service.cc:342] P d3a480e831d649139ff016fb68aee3ee: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-a7ef86a50a514e8ba09c76ca674b2100 received from {username='slave'} at 127.0.0.1:48202
I20240426 09:54:54.098985 10100 tablet_copy_service.cc:433] P d3a480e831d649139ff016fb68aee3ee: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-a7ef86a50a514e8ba09c76ca674b2100 on tablet a7ef86a50a514e8ba09c76ca674b2100 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:54:54.122907 10620 raft_consensus.cc:1203] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 09:54:54.235138 10620 raft_consensus.cc:1203] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 09:54:54.450348 10620 raft_consensus.cc:1203] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 09:54:54.482232 10620 raft_consensus.cc:1203] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 09:54:54.659377 10702 raft_consensus.cc:1050] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:54:54.661953 10702 consensus_queue.cc:237] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:54:54.668267 10620 raft_consensus.cc:1261] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:54.669517 10658 consensus_queue.cc:1035] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:54.670378 9940 raft_consensus.cc:1261] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:54.671461 10090 raft_consensus.cc:1261] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:54.672909 10658 consensus_queue.cc:1035] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:54.675141 10658 consensus_queue.cc:1035] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:54.681295 10702 raft_consensus.cc:2932] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.682564 9940 raft_consensus.cc:2932] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.689657 10090 raft_consensus.cc:2932] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.691725 10620 raft_consensus.cc:2932] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.700327 9836 catalog_manager.cc:5484] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:54.733721 10014 consensus_queue.cc:237] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:54:54.740240 10620 raft_consensus.cc:1261] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:54:54.740753 10086 raft_consensus.cc:1261] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:54:54.741355 10702 consensus_queue.cc:1035] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 09:54:54.743324 10653 consensus_queue.cc:1035] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:54.752094 10653 raft_consensus.cc:2932] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.757164 10090 raft_consensus.cc:2932] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.757035 10620 raft_consensus.cc:2932] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.773873 9835 catalog_manager.cc:5484] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.783751 9925 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c0fe10a2b3746f9909d3596580b6854 with delete_type TABLET_DATA_TOMBSTONED (TS f12521b6d51242629cf4297864df00b9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47232
I20240426 09:54:54.785804 10712 tablet_replica.cc:330] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9: stopping tablet replica
I20240426 09:54:54.786381 9825 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5c0fe10a2b3746f9909d3596580b6854 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:54:54.786664 10712 raft_consensus.cc:2229] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:54.787392 10712 raft_consensus.cc:2258] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:54.796013 10712 ts_tablet_manager.cc:1875] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:54:54.812975 10712 ts_tablet_manager.cc:1888] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 09:54:54.813493 10712 log.cc:1196] T 5c0fe10a2b3746f9909d3596580b6854 P f12521b6d51242629cf4297864df00b9: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-0-root/wals/5c0fe10a2b3746f9909d3596580b6854
I20240426 09:54:54.815109 9826 catalog_manager.cc:4830] TS f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): tablet 5c0fe10a2b3746f9909d3596580b6854 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:54:54.856467 10658 raft_consensus.cc:1050] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:54:54.858330 10658 consensus_queue.cc:237] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:54:54.863530 10620 raft_consensus.cc:1261] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:54.863734 10090 raft_consensus.cc:1261] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:54.864713 10702 consensus_queue.cc:1035] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:54.866024 9940 raft_consensus.cc:1261] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:54.866330 10658 consensus_queue.cc:1035] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20240426 09:54:54.869858 10702 consensus_queue.cc:1035] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:54.881184 10653 raft_consensus.cc:2932] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.881647 9940 raft_consensus.cc:2932] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.886947 10090 raft_consensus.cc:2932] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.891247 10620 raft_consensus.cc:2932] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.903823 9835 catalog_manager.cc:5484] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.907193 10672 raft_consensus.cc:1050] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:54:54.909101 10672 consensus_queue.cc:237] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:54:54.925635 10011 raft_consensus.cc:1261] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:54.925665 10620 raft_consensus.cc:1261] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:54.926981 10672 consensus_queue.cc:1035] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:54.927788 10711 consensus_queue.cc:1035] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:54.924212 10014 consensus_queue.cc:237] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:54:54.938292 10672 raft_consensus.cc:2932] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.942560 10011 raft_consensus.cc:2932] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.949009 10619 raft_consensus.cc:2932] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.950402 9938 raft_consensus.cc:1261] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:54:54.952327 10653 consensus_queue.cc:1035] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:54.949069 10620 raft_consensus.cc:1261] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:54:54.954939 9940 raft_consensus.cc:1261] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:54.958735 10653 consensus_queue.cc:1035] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:54.960649 10572 consensus_queue.cc:1035] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240426 09:54:54.961661 9839 catalog_manager.cc:5484] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:54:54.972146 10717 raft_consensus.cc:2932] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.973634 9940 raft_consensus.cc:2932] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.979578 10620 raft_consensus.cc:2932] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.988306 9938 raft_consensus.cc:2932] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:54.995537 9825 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ddc5aa33dc824544ade18161bf2800a6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:54:54.998733 9835 catalog_manager.cc:5484] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 reported cstate change: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:55.015002 10077 tablet_service.cc:1514] Processing DeleteTablet for tablet ddc5aa33dc824544ade18161bf2800a6 with delete_type TABLET_DATA_TOMBSTONED (TS d3a480e831d649139ff016fb68aee3ee not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59730
I20240426 09:54:55.037262 10722 tablet_replica.cc:330] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee: stopping tablet replica
I20240426 09:54:55.038367 10722 raft_consensus.cc:2229] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:55.040369 10722 raft_consensus.cc:2258] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:55.046941 10722 ts_tablet_manager.cc:1875] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:54:55.053300 10653 raft_consensus.cc:1050] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:54:55.055595 10653 consensus_queue.cc:237] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:54:55.063081 10620 raft_consensus.cc:1261] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:55.063500 9938 raft_consensus.cc:1261] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:55.070556 10090 raft_consensus.cc:1261] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:55.071848 10702 consensus_queue.cc:1035] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:55.074265 10653 consensus_queue.cc:1035] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:55.075469 10717 consensus_queue.cc:1035] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:55.097663 10722 ts_tablet_manager.cc:1888] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 09:54:55.098068 10722 log.cc:1196] T ddc5aa33dc824544ade18161bf2800a6 P d3a480e831d649139ff016fb68aee3ee: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-2-root/wals/ddc5aa33dc824544ade18161bf2800a6
I20240426 09:54:55.099617 9823 catalog_manager.cc:4830] TS d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): tablet ddc5aa33dc824544ade18161bf2800a6 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:54:55.099186 10653 raft_consensus.cc:2932] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:55.102252 9938 raft_consensus.cc:2932] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:55.115696 10086 raft_consensus.cc:2932] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:55.126489 9838 catalog_manager.cc:5484] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:55.132164 10620 raft_consensus.cc:2932] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:55.154098 10013 consensus_queue.cc:237] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:54:55.159062 10620 raft_consensus.cc:1261] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:54:55.159184 9940 raft_consensus.cc:1261] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:54:55.160251 10717 consensus_queue.cc:1035] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:55.161005 10653 consensus_queue.cc:1035] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:55.167294 10717 raft_consensus.cc:2932] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:55.168913 9938 raft_consensus.cc:2932] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:55.174649 9825 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7b63eef1eb5e4fabb35fb66b0f5a41b4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:54:55.174876 10620 raft_consensus.cc:2932] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:55.186365 9835 catalog_manager.cc:5484] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:54:55.203965 10077 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b63eef1eb5e4fabb35fb66b0f5a41b4 with delete_type TABLET_DATA_TOMBSTONED (TS d3a480e831d649139ff016fb68aee3ee not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59730
I20240426 09:54:55.204818 10722 tablet_replica.cc:330] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee: stopping tablet replica
I20240426 09:54:55.205539 10722 raft_consensus.cc:2229] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:55.206039 10722 raft_consensus.cc:2258] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:55.208665 10722 ts_tablet_manager.cc:1875] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:54:55.222194 10722 ts_tablet_manager.cc:1888] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 09:54:55.222501 10722 log.cc:1196] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P d3a480e831d649139ff016fb68aee3ee: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-2-root/wals/7b63eef1eb5e4fabb35fb66b0f5a41b4
I20240426 09:54:55.225549 9823 catalog_manager.cc:4830] TS d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): tablet 7b63eef1eb5e4fabb35fb66b0f5a41b4 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:54:55.459519 10086 consensus_queue.cc:237] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:54:55.463174 10620 raft_consensus.cc:1261] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:54:55.463356 9936 raft_consensus.cc:1261] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:54:55.464558 10672 consensus_queue.cc:1035] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:55.465257 10711 consensus_queue.cc:1035] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:55.471251 10672 raft_consensus.cc:2932] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:55.472522 9936 raft_consensus.cc:2932] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:55.479609 10620 raft_consensus.cc:2932] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:55.487697 9823 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a7ef86a50a514e8ba09c76ca674b2100 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:54:55.488704 9835 catalog_manager.cc:5484] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 reported cstate change: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:55.498739 10006 tablet_service.cc:1514] Processing DeleteTablet for tablet a7ef86a50a514e8ba09c76ca674b2100 with delete_type TABLET_DATA_TOMBSTONED (TS 826f03b2fb9c4b9483623b041a1bf80e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34362
I20240426 09:54:55.504101 10727 tablet_replica.cc:330] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e: stopping tablet replica
I20240426 09:54:55.504880 10727 raft_consensus.cc:2229] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:55.505332 10727 raft_consensus.cc:2258] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:55.508405 10727 ts_tablet_manager.cc:1875] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:54:55.520011 10727 ts_tablet_manager.cc:1888] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 09:54:55.520275 10727 log.cc:1196] T a7ef86a50a514e8ba09c76ca674b2100 P 826f03b2fb9c4b9483623b041a1bf80e: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-1-root/wals/a7ef86a50a514e8ba09c76ca674b2100
I20240426 09:54:55.521288 9825 catalog_manager.cc:4830] TS 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): tablet a7ef86a50a514e8ba09c76ca674b2100 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:54:56.550393 10013 consensus_queue.cc:237] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
W20240426 09:54:56.555497 9980 consensus_peers.cc:485] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e11b7521039c4feba17dc65ac6c819fb. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:56.555619 9940 raft_consensus.cc:1261] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:54:56.556887 10090 raft_consensus.cc:1261] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:54:56.556825 10717 consensus_queue.cc:1035] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:56.558323 10653 consensus_queue.cc:1035] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:56.563213 10717 raft_consensus.cc:2932] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:56.564361 9940 raft_consensus.cc:2932] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:56.565688 10090 raft_consensus.cc:2932] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:56.580899 9839 catalog_manager.cc:5484] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:56.615386 9940 consensus_queue.cc:237] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:56.627535 10090 raft_consensus.cc:1261] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:54:56.628893 10193 consensus_queue.cc:1035] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240426 09:54:56.654989 9905 consensus_peers.cc:485] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6ed47e1980f4a14a46cc61ea156eb53. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:56.658185 10013 raft_consensus.cc:1261] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:54:56.659340 10193 consensus_queue.cc:1035] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:56.664186 10738 raft_consensus.cc:2932] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:56.665612 10086 raft_consensus.cc:2932] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:56.679116 10014 raft_consensus.cc:2932] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:56.684607 10086 consensus_queue.cc:237] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:56.690733 9835 catalog_manager.cc:5484] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:56.700584 10011 raft_consensus.cc:1261] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:54:56.702087 10672 consensus_queue.cc:1035] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:56.702893 9940 raft_consensus.cc:1261] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 09:54:56.702970 10054 consensus_peers.cc:485] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fc982aface54033a68641d8d809fd3b. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:56.704493 10672 consensus_queue.cc:1035] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:56.711248 10746 raft_consensus.cc:2932] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:56.712288 9940 raft_consensus.cc:2932] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:56.723660 10011 raft_consensus.cc:2932] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:56.732316 10014 consensus_queue.cc:237] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:56.733366 9838 catalog_manager.cc:5484] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 09:54:56.742542 10086 raft_consensus.cc:1261] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 09:54:56.744385 9980 consensus_peers.cc:485] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbe8813b366b4644872fbec0addf4443. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:56.744624 9938 raft_consensus.cc:1261] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:54:56.744947 10732 consensus_queue.cc:1035] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:56.747130 10732 consensus_queue.cc:1035] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:56.752601 10653 raft_consensus.cc:2932] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:56.754015 10086 raft_consensus.cc:2932] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:56.775202 9938 raft_consensus.cc:2932] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:56.808027 9835 catalog_manager.cc:5484] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:56.987607 10753 ts_tablet_manager.cc:902] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:56.995335 10753 tablet_copy_client.cc:308] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.130:44411
I20240426 09:54:56.996693 10026 tablet_copy_service.cc:140] P 826f03b2fb9c4b9483623b041a1bf80e: Received BeginTabletCopySession request for tablet e11b7521039c4feba17dc65ac6c819fb from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:36762)
I20240426 09:54:56.997216 10026 tablet_copy_service.cc:161] P 826f03b2fb9c4b9483623b041a1bf80e: Beginning new tablet copy session on tablet e11b7521039c4feba17dc65ac6c819fb from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:36762: session id = 1fee3da01ebc463396397057fdb81a68-e11b7521039c4feba17dc65ac6c819fb
I20240426 09:54:57.003608 10026 tablet_copy_source_session.cc:205] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:54:57.007210 10753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11b7521039c4feba17dc65ac6c819fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:57.020570 10753 tablet_copy_client.cc:783] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:54:57.021034 10753 tablet_copy_client.cc:647] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:54:57.024801 10753 tablet_copy_client.cc:523] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:54:57.031600 10753 tablet_bootstrap.cc:492] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:54:57.048383 10753 tablet_bootstrap.cc:492] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:54:57.049141 10753 tablet_bootstrap.cc:492] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:54:57.049662 10753 ts_tablet_manager.cc:1367] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20240426 09:54:57.052109 10753 raft_consensus.cc:348] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:57.052716 10753 raft_consensus.cc:729] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:54:57.053251 10753 consensus_queue.cc:260] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:57.056077 10753 ts_tablet_manager.cc:1398] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20240426 09:54:57.057942 10026 tablet_copy_service.cc:342] P 826f03b2fb9c4b9483623b041a1bf80e: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-e11b7521039c4feba17dc65ac6c819fb received from {username='slave'} at 127.0.0.1:36762
I20240426 09:54:57.058326 10026 tablet_copy_service.cc:433] P 826f03b2fb9c4b9483623b041a1bf80e: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-e11b7521039c4feba17dc65ac6c819fb on tablet e11b7521039c4feba17dc65ac6c819fb with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:54:57.174031 10753 ts_tablet_manager.cc:902] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:54:57.203580 10753 tablet_copy_client.cc:308] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.131:34821
I20240426 09:54:57.205410 10100 tablet_copy_service.cc:140] P d3a480e831d649139ff016fb68aee3ee: Received BeginTabletCopySession request for tablet 5fc982aface54033a68641d8d809fd3b from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:48202)
I20240426 09:54:57.205937 10100 tablet_copy_service.cc:161] P d3a480e831d649139ff016fb68aee3ee: Beginning new tablet copy session on tablet 5fc982aface54033a68641d8d809fd3b from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:48202: session id = 1fee3da01ebc463396397057fdb81a68-5fc982aface54033a68641d8d809fd3b
I20240426 09:54:57.214825 10100 tablet_copy_source_session.cc:205] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:54:57.217859 10753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fc982aface54033a68641d8d809fd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:57.218001 10764 ts_tablet_manager.cc:902] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113)
I20240426 09:54:57.236398 10766 ts_tablet_manager.cc:902] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:54:57.238935 10753 tablet_copy_client.cc:783] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:54:57.239488 10753 tablet_copy_client.cc:647] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:54:57.245426 10753 tablet_copy_client.cc:523] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:54:57.262152 10764 tablet_copy_client.cc:308] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.129:38113
I20240426 09:54:57.264652 10753 tablet_bootstrap.cc:492] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:54:57.287845 10766 tablet_copy_client.cc:308] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.130:44411
I20240426 09:54:57.289382 10026 tablet_copy_service.cc:140] P 826f03b2fb9c4b9483623b041a1bf80e: Received BeginTabletCopySession request for tablet bbe8813b366b4644872fbec0addf4443 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:36762)
I20240426 09:54:57.289921 10026 tablet_copy_service.cc:161] P 826f03b2fb9c4b9483623b041a1bf80e: Beginning new tablet copy session on tablet bbe8813b366b4644872fbec0addf4443 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:36762: session id = 1fee3da01ebc463396397057fdb81a68-bbe8813b366b4644872fbec0addf4443
I20240426 09:54:57.291464 10753 tablet_bootstrap.cc:492] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:54:57.292346 10753 tablet_bootstrap.cc:492] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:54:57.293015 10753 ts_tablet_manager.cc:1367] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.029s user 0.026s sys 0.000s
I20240426 09:54:57.295651 10753 raft_consensus.cc:348] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:57.296547 10753 raft_consensus.cc:729] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:54:57.296952 10026 tablet_copy_source_session.cc:205] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:54:57.297210 10753 consensus_queue.cc:260] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:57.298611 9951 tablet_copy_service.cc:140] P f12521b6d51242629cf4297864df00b9: Received BeginTabletCopySession request for tablet c6ed47e1980f4a14a46cc61ea156eb53 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:40928)
I20240426 09:54:57.300233 9951 tablet_copy_service.cc:161] P f12521b6d51242629cf4297864df00b9: Beginning new tablet copy session on tablet c6ed47e1980f4a14a46cc61ea156eb53 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:40928: session id = 1fee3da01ebc463396397057fdb81a68-c6ed47e1980f4a14a46cc61ea156eb53
I20240426 09:54:57.304742 10753 ts_tablet_manager.cc:1398] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240426 09:54:57.305001 10766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe8813b366b4644872fbec0addf4443. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:57.310330 9951 tablet_copy_source_session.cc:205] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:54:57.322233 10764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ed47e1980f4a14a46cc61ea156eb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:54:57.332950 10100 tablet_copy_service.cc:342] P d3a480e831d649139ff016fb68aee3ee: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-5fc982aface54033a68641d8d809fd3b received from {username='slave'} at 127.0.0.1:48202
I20240426 09:54:57.339331 10100 tablet_copy_service.cc:433] P d3a480e831d649139ff016fb68aee3ee: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-5fc982aface54033a68641d8d809fd3b on tablet 5fc982aface54033a68641d8d809fd3b with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:54:57.347931 10766 tablet_copy_client.cc:783] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:54:57.348610 10766 tablet_copy_client.cc:647] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:54:57.351516 10764 tablet_copy_client.cc:783] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:54:57.352090 10764 tablet_copy_client.cc:647] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:54:57.352350 10766 tablet_copy_client.cc:523] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:54:57.355340 10764 tablet_copy_client.cc:523] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:54:57.360555 10766 tablet_bootstrap.cc:492] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:54:57.363332 10764 tablet_bootstrap.cc:492] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:54:57.386870 10766 tablet_bootstrap.cc:492] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:54:57.389092 10766 tablet_bootstrap.cc:492] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:54:57.389748 10766 ts_tablet_manager.cc:1367] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.016s
I20240426 09:54:57.390836 10764 tablet_bootstrap.cc:492] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:54:57.391775 10764 tablet_bootstrap.cc:492] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:54:57.392405 10764 ts_tablet_manager.cc:1367] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.029s user 0.021s sys 0.005s
I20240426 09:54:57.392735 10766 raft_consensus.cc:348] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:57.393618 10766 raft_consensus.cc:729] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:54:57.394220 10766 consensus_queue.cc:260] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:57.396811 10766 ts_tablet_manager.cc:1398] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240426 09:54:57.397308 10764 raft_consensus.cc:348] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:57.398052 10764 raft_consensus.cc:729] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:54:57.398221 10026 tablet_copy_service.cc:342] P 826f03b2fb9c4b9483623b041a1bf80e: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-bbe8813b366b4644872fbec0addf4443 received from {username='slave'} at 127.0.0.1:36762
I20240426 09:54:57.398619 10026 tablet_copy_service.cc:433] P 826f03b2fb9c4b9483623b041a1bf80e: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-bbe8813b366b4644872fbec0addf4443 on tablet bbe8813b366b4644872fbec0addf4443 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:54:57.398707 10764 consensus_queue.cc:260] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:57.402211 10764 ts_tablet_manager.cc:1398] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240426 09:54:57.406684 9951 tablet_copy_service.cc:342] P f12521b6d51242629cf4297864df00b9: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-c6ed47e1980f4a14a46cc61ea156eb53 received from {username='slave'} at 127.0.0.1:40928
I20240426 09:54:57.407099 9951 tablet_copy_service.cc:433] P f12521b6d51242629cf4297864df00b9: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-c6ed47e1980f4a14a46cc61ea156eb53 on tablet c6ed47e1980f4a14a46cc61ea156eb53 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:54:57.554306 10619 raft_consensus.cc:1203] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 09:54:57.611680 10619 raft_consensus.cc:1203] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 09:54:57.611963 10620 raft_consensus.cc:1203] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 09:54:57.635994 10620 raft_consensus.cc:1203] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 09:54:58.071765 10782 raft_consensus.cc:1050] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:54:58.074719 10782 consensus_queue.cc:237] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:54:58.088232 10089 raft_consensus.cc:1261] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:58.088238 10620 raft_consensus.cc:1261] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:58.088711 10011 raft_consensus.cc:1261] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:58.090271 10676 consensus_queue.cc:1035] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:58.091864 10676 consensus_queue.cc:1035] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:58.093945 10193 consensus_queue.cc:1035] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:58.111323 10653 raft_consensus.cc:1050] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:54:58.116950 10653 consensus_queue.cc:237] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:54:58.117473 10676 raft_consensus.cc:2932] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.119038 10619 raft_consensus.cc:2932] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.123332 10089 raft_consensus.cc:2932] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.130184 10711 raft_consensus.cc:1050] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:54:58.132916 10711 consensus_queue.cc:237] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:54:58.135788 10619 raft_consensus.cc:1261] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:58.137187 10086 raft_consensus.cc:1261] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:58.137529 10780 consensus_queue.cc:1035] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 09:54:58.138343 9938 raft_consensus.cc:1261] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:58.135380 9838 catalog_manager.cc:5484] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.138520 10733 consensus_queue.cc:1035] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:58.139390 10653 consensus_queue.cc:1035] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:58.159242 10014 raft_consensus.cc:2932] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.179234 10620 raft_consensus.cc:1261] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:58.181154 10799 consensus_queue.cc:1035] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 09:54:58.185468 10653 raft_consensus.cc:1050] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:54:58.189400 10653 consensus_queue.cc:237] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:54:58.194784 10011 raft_consensus.cc:1261] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:58.196058 10799 consensus_queue.cc:1035] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:58.219071 10618 raft_consensus.cc:1261] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:58.220893 10653 consensus_queue.cc:1035] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:58.220888 10089 raft_consensus.cc:1261] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:58.225741 10653 consensus_queue.cc:1035] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:58.227097 10799 raft_consensus.cc:2932] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.231791 10780 raft_consensus.cc:2932] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
W20240426 09:54:58.227124 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 93827.4us
I20240426 09:54:58.241147 9938 raft_consensus.cc:1261] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:58.241786 10011 raft_consensus.cc:2932] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.243113 10090 raft_consensus.cc:2932] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.244102 9938 raft_consensus.cc:2932] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.240800 10619 raft_consensus.cc:2932] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.252701 10620 raft_consensus.cc:2932] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.257416 9939 raft_consensus.cc:1261] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:54:58.286470 10653 raft_consensus.cc:2932] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.282101 9836 catalog_manager.cc:5484] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:54:58.295750 10619 raft_consensus.cc:2932] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.296499 9836 catalog_manager.cc:5484] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.302016 10733 consensus_queue.cc:1035] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:58.302948 9938 consensus_queue.cc:237] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:54:58.304371 10089 raft_consensus.cc:2932] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.310931 10799 consensus_queue.cc:1035] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:54:58.314500 10620 raft_consensus.cc:1261] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:54:58.321264 10793 consensus_queue.cc:1035] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 09:54:58.333261 10193 raft_consensus.cc:2932] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.337954 9940 raft_consensus.cc:2932] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.339509 10620 raft_consensus.cc:2932] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.341847 9826 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6ed47e1980f4a14a46cc61ea156eb53 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:54:58.349262 10089 raft_consensus.cc:1261] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:54:58.351950 9835 catalog_manager.cc:5484] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.354594 10014 consensus_queue.cc:237] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:54:58.365746 10193 consensus_queue.cc:1035] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:58.367236 10089 raft_consensus.cc:1261] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:54:58.368027 10620 raft_consensus.cc:1261] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:54:58.369453 10780 consensus_queue.cc:1035] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:58.370388 10653 consensus_queue.cc:1035] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:58.372481 10086 raft_consensus.cc:2932] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.379503 9938 raft_consensus.cc:2932] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.386732 10733 raft_consensus.cc:2932] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.390671 9835 catalog_manager.cc:5484] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.394665 10011 consensus_queue.cc:237] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:54:58.400216 10086 raft_consensus.cc:2932] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.405234 10620 raft_consensus.cc:1261] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:54:58.407518 10090 raft_consensus.cc:1261] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:54:58.410375 10006 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ed47e1980f4a14a46cc61ea156eb53 with delete_type TABLET_DATA_TOMBSTONED (TS 826f03b2fb9c4b9483623b041a1bf80e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34362
I20240426 09:54:58.407712 10806 consensus_queue.cc:1035] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20240426 09:54:58.416736 9839 catalog_manager.cc:5484] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.418609 10653 consensus_queue.cc:1035] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 09:54:58.420663 10619 raft_consensus.cc:2932] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.424880 9825 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bbe8813b366b4644872fbec0addf4443 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:54:58.434105 9925 tablet_service.cc:1514] Processing DeleteTablet for tablet bbe8813b366b4644872fbec0addf4443 with delete_type TABLET_DATA_TOMBSTONED (TS f12521b6d51242629cf4297864df00b9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47232
I20240426 09:54:58.439980 10810 tablet_replica.cc:330] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e: stopping tablet replica
I20240426 09:54:58.440819 10813 tablet_replica.cc:330] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9: stopping tablet replica
I20240426 09:54:58.445924 10810 raft_consensus.cc:2229] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:58.448966 10813 raft_consensus.cc:2229] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:58.454697 10810 raft_consensus.cc:2258] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:58.454972 10813 raft_consensus.cc:2258] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:58.445377 10806 raft_consensus.cc:2932] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.458297 10810 ts_tablet_manager.cc:1875] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:54:58.458783 10813 ts_tablet_manager.cc:1875] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:54:58.482021 10619 raft_consensus.cc:2932] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.495224 9839 catalog_manager.cc:5484] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.502875 10813 ts_tablet_manager.cc:1888] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 09:54:58.503468 10813 log.cc:1196] T bbe8813b366b4644872fbec0addf4443 P f12521b6d51242629cf4297864df00b9: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-0-root/wals/bbe8813b366b4644872fbec0addf4443
I20240426 09:54:58.505242 9826 catalog_manager.cc:4830] TS f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): tablet bbe8813b366b4644872fbec0addf4443 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:54:58.506064 10089 raft_consensus.cc:2932] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.507328 10810 ts_tablet_manager.cc:1888] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 09:54:58.507805 10810 log.cc:1196] T c6ed47e1980f4a14a46cc61ea156eb53 P 826f03b2fb9c4b9483623b041a1bf80e: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-1-root/wals/c6ed47e1980f4a14a46cc61ea156eb53
I20240426 09:54:58.509768 9825 catalog_manager.cc:4830] TS 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): tablet c6ed47e1980f4a14a46cc61ea156eb53 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:54:58.523715 9925 tablet_service.cc:1514] Processing DeleteTablet for tablet e11b7521039c4feba17dc65ac6c819fb with delete_type TABLET_DATA_TOMBSTONED (TS f12521b6d51242629cf4297864df00b9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47232
I20240426 09:54:58.523855 9825 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e11b7521039c4feba17dc65ac6c819fb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:54:58.525226 10813 tablet_replica.cc:330] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9: stopping tablet replica
I20240426 09:54:58.526160 10813 raft_consensus.cc:2229] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:58.527030 10813 raft_consensus.cc:2258] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:58.531751 10813 ts_tablet_manager.cc:1875] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:54:58.544768 10813 ts_tablet_manager.cc:1888] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 09:54:58.545115 10813 log.cc:1196] T e11b7521039c4feba17dc65ac6c819fb P f12521b6d51242629cf4297864df00b9: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-0-root/wals/e11b7521039c4feba17dc65ac6c819fb
I20240426 09:54:58.546357 9826 catalog_manager.cc:4830] TS f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): tablet e11b7521039c4feba17dc65ac6c819fb (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:54:58.792205 10089 consensus_queue.cc:237] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:54:58.795979 10619 raft_consensus.cc:1261] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:54:58.797004 10796 consensus_queue.cc:1035] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:58.797565 9938 raft_consensus.cc:1261] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:54:58.799106 10749 consensus_queue.cc:1035] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:54:58.803706 10749 raft_consensus.cc:2932] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.808012 10619 raft_consensus.cc:2932] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.812464 9935 raft_consensus.cc:2932] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.821885 9838 catalog_manager.cc:5484] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:54:58.825431 9823 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5fc982aface54033a68641d8d809fd3b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:54:58.831800 10006 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fc982aface54033a68641d8d809fd3b with delete_type TABLET_DATA_TOMBSTONED (TS 826f03b2fb9c4b9483623b041a1bf80e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34362
I20240426 09:54:58.835080 10810 tablet_replica.cc:330] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e: stopping tablet replica
I20240426 09:54:58.836124 10810 raft_consensus.cc:2229] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:54:58.836727 10810 raft_consensus.cc:2258] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:54:58.842285 10810 ts_tablet_manager.cc:1875] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:54:58.863085 10810 ts_tablet_manager.cc:1888] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 09:54:58.863359 10810 log.cc:1196] T 5fc982aface54033a68641d8d809fd3b P 826f03b2fb9c4b9483623b041a1bf80e: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-1-root/wals/5fc982aface54033a68641d8d809fd3b
I20240426 09:54:58.864460 9825 catalog_manager.cc:4830] TS 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): tablet 5fc982aface54033a68641d8d809fd3b (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:54:59.906912 9935 consensus_queue.cc:237] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:59.911455 10086 raft_consensus.cc:1261] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:54:59.912494 10014 raft_consensus.cc:1261] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 09:54:59.916208 9905 consensus_peers.cc:485] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09b98bf14ce848eca1c7ca1840a516b1. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:59.916703 10816 consensus_queue.cc:1035] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:59.917757 10794 consensus_queue.cc:1035] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:59.924793 10819 raft_consensus.cc:2932] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:59.926424 10014 raft_consensus.cc:2932] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:59.926429 10086 raft_consensus.cc:2932] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:59.938753 9838 catalog_manager.cc:5484] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:59.942787 9935 consensus_queue.cc:237] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:54:59.951898 10086 raft_consensus.cc:1261] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:54:59.952816 10816 consensus_queue.cc:1035] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:59.953223 10014 raft_consensus.cc:1261] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 09:54:59.953707 9905 consensus_peers.cc:485] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 951e4c48fcb642e58ab79652d3f86261. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:54:59.954406 10819 consensus_queue.cc:1035] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:54:59.963265 10086 raft_consensus.cc:2932] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:59.961750 10816 raft_consensus.cc:2932] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:59.970630 10014 raft_consensus.cc:2932] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:59.984328 9835 catalog_manager.cc:5484] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:54:59.989569 9935 consensus_queue.cc:237] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
W20240426 09:55:00.003417 9905 consensus_peers.cc:485] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c21d6dedd1441bd977f09cdcf41d436. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:00.005635 10086 raft_consensus.cc:1261] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:00.006461 10010 raft_consensus.cc:1261] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:00.007228 10816 consensus_queue.cc:1035] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:00.008108 10820 consensus_queue.cc:1035] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:00.016675 10816 raft_consensus.cc:2932] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:00.018275 10010 raft_consensus.cc:2932] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:00.020401 10086 raft_consensus.cc:2932] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:00.028682 10089 consensus_queue.cc:237] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:00.035487 9835 catalog_manager.cc:5484] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 2 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:00.047283 9935 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:55:00.048100 10010 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 09:55:00.048007 10054 consensus_peers.cc:485] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51125fbde06b4a588fb6171dec56a605. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:00.050345 10749 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:00.051127 10799 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:00.059410 10749 raft_consensus.cc:2932] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:00.060734 10010 raft_consensus.cc:2932] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:00.065764 9935 raft_consensus.cc:2932] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:00.074648 9835 catalog_manager.cc:5484] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:00.363543 10839 ts_tablet_manager.cc:902] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113)
I20240426 09:55:00.367662 10839 tablet_copy_client.cc:308] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.129:38113
I20240426 09:55:00.369508 9951 tablet_copy_service.cc:140] P f12521b6d51242629cf4297864df00b9: Received BeginTabletCopySession request for tablet 951e4c48fcb642e58ab79652d3f86261 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:40928)
I20240426 09:55:00.370076 9951 tablet_copy_service.cc:161] P f12521b6d51242629cf4297864df00b9: Beginning new tablet copy session on tablet 951e4c48fcb642e58ab79652d3f86261 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:40928: session id = 1fee3da01ebc463396397057fdb81a68-951e4c48fcb642e58ab79652d3f86261
I20240426 09:55:00.376178 9951 tablet_copy_source_session.cc:205] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:00.378641 10839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 951e4c48fcb642e58ab79652d3f86261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:00.390560 10839 tablet_copy_client.cc:783] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:00.391105 10839 tablet_copy_client.cc:647] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:00.394891 10839 tablet_copy_client.cc:523] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:00.402007 10839 tablet_bootstrap.cc:492] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:00.428836 10839 tablet_bootstrap.cc:492] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:00.429733 10839 tablet_bootstrap.cc:492] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:00.430301 10839 ts_tablet_manager.cc:1367] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.029s user 0.022s sys 0.005s
I20240426 09:55:00.432998 10839 raft_consensus.cc:348] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:00.433717 10839 raft_consensus.cc:729] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:00.434304 10839 consensus_queue.cc:260] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:00.437443 10839 ts_tablet_manager.cc:1398] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20240426 09:55:00.439983 9951 tablet_copy_service.cc:342] P f12521b6d51242629cf4297864df00b9: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-951e4c48fcb642e58ab79652d3f86261 received from {username='slave'} at 127.0.0.1:40928
I20240426 09:55:00.440367 9951 tablet_copy_service.cc:433] P f12521b6d51242629cf4297864df00b9: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-951e4c48fcb642e58ab79652d3f86261 on tablet 951e4c48fcb642e58ab79652d3f86261 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:00.477685 10839 ts_tablet_manager.cc:902] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:00.481070 10847 ts_tablet_manager.cc:902] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113)
I20240426 09:55:00.487349 10839 tablet_copy_client.cc:308] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.131:34821
I20240426 09:55:00.489341 10100 tablet_copy_service.cc:140] P d3a480e831d649139ff016fb68aee3ee: Received BeginTabletCopySession request for tablet 51125fbde06b4a588fb6171dec56a605 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:48202)
I20240426 09:55:00.489887 10100 tablet_copy_service.cc:161] P d3a480e831d649139ff016fb68aee3ee: Beginning new tablet copy session on tablet 51125fbde06b4a588fb6171dec56a605 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:48202: session id = 1fee3da01ebc463396397057fdb81a68-51125fbde06b4a588fb6171dec56a605
I20240426 09:55:00.496277 10100 tablet_copy_source_session.cc:205] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:00.499020 10839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51125fbde06b4a588fb6171dec56a605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:00.507004 10847 tablet_copy_client.cc:308] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.129:38113
I20240426 09:55:00.508566 9951 tablet_copy_service.cc:140] P f12521b6d51242629cf4297864df00b9: Received BeginTabletCopySession request for tablet 09b98bf14ce848eca1c7ca1840a516b1 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:40928)
I20240426 09:55:00.509048 9951 tablet_copy_service.cc:161] P f12521b6d51242629cf4297864df00b9: Beginning new tablet copy session on tablet 09b98bf14ce848eca1c7ca1840a516b1 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:40928: session id = 1fee3da01ebc463396397057fdb81a68-09b98bf14ce848eca1c7ca1840a516b1
I20240426 09:55:00.514752 10839 tablet_copy_client.cc:783] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:00.515324 10839 tablet_copy_client.cc:647] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:00.519696 9951 tablet_copy_source_session.cc:205] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:00.523798 10855 ts_tablet_manager.cc:902] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113)
I20240426 09:55:00.523712 10847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b98bf14ce848eca1c7ca1840a516b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:00.525463 10839 tablet_copy_client.cc:523] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:00.541477 10839 tablet_bootstrap.cc:492] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:00.544410 10847 tablet_copy_client.cc:783] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:00.545042 10847 tablet_copy_client.cc:647] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:00.549777 10847 tablet_copy_client.cc:523] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:00.558530 10847 tablet_bootstrap.cc:492] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:00.559386 10855 tablet_copy_client.cc:308] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.129:38113
I20240426 09:55:00.560932 9951 tablet_copy_service.cc:140] P f12521b6d51242629cf4297864df00b9: Received BeginTabletCopySession request for tablet 1c21d6dedd1441bd977f09cdcf41d436 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:40928)
I20240426 09:55:00.561424 9951 tablet_copy_service.cc:161] P f12521b6d51242629cf4297864df00b9: Beginning new tablet copy session on tablet 1c21d6dedd1441bd977f09cdcf41d436 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:40928: session id = 1fee3da01ebc463396397057fdb81a68-1c21d6dedd1441bd977f09cdcf41d436
I20240426 09:55:00.582749 9951 tablet_copy_source_session.cc:205] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:00.605077 10839 tablet_bootstrap.cc:492] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:00.605957 10839 tablet_bootstrap.cc:492] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:00.607420 10839 ts_tablet_manager.cc:1367] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.066s user 0.017s sys 0.029s
I20240426 09:55:00.608002 10855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c21d6dedd1441bd977f09cdcf41d436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:00.609961 10839 raft_consensus.cc:348] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:00.612404 10839 raft_consensus.cc:729] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:00.613061 10839 consensus_queue.cc:260] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:00.615666 10839 ts_tablet_manager.cc:1398] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20240426 09:55:00.617060 10100 tablet_copy_service.cc:342] P d3a480e831d649139ff016fb68aee3ee: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-51125fbde06b4a588fb6171dec56a605 received from {username='slave'} at 127.0.0.1:48202
I20240426 09:55:00.617417 10100 tablet_copy_service.cc:433] P d3a480e831d649139ff016fb68aee3ee: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-51125fbde06b4a588fb6171dec56a605 on tablet 51125fbde06b4a588fb6171dec56a605 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:00.622054 10847 tablet_bootstrap.cc:492] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:00.625877 10847 tablet_bootstrap.cc:492] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:00.626480 10847 ts_tablet_manager.cc:1367] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.068s user 0.027s sys 0.032s
I20240426 09:55:00.629087 10847 raft_consensus.cc:348] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:00.629931 10847 raft_consensus.cc:729] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:00.630546 10847 consensus_queue.cc:260] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:00.632499 10847 ts_tablet_manager.cc:1398] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240426 09:55:00.634323 9951 tablet_copy_service.cc:342] P f12521b6d51242629cf4297864df00b9: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-09b98bf14ce848eca1c7ca1840a516b1 received from {username='slave'} at 127.0.0.1:40928
I20240426 09:55:00.634629 9951 tablet_copy_service.cc:433] P f12521b6d51242629cf4297864df00b9: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-09b98bf14ce848eca1c7ca1840a516b1 on tablet 09b98bf14ce848eca1c7ca1840a516b1 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:00.641584 10855 tablet_copy_client.cc:783] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:00.642297 10855 tablet_copy_client.cc:647] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:00.645731 10855 tablet_copy_client.cc:523] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:00.656176 10855 tablet_bootstrap.cc:492] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:00.676990 10855 tablet_bootstrap.cc:492] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:00.677598 10855 tablet_bootstrap.cc:492] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:00.678187 10855 ts_tablet_manager.cc:1367] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.000s
I20240426 09:55:00.680812 10855 raft_consensus.cc:348] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:00.681715 10855 raft_consensus.cc:729] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:00.682348 10855 consensus_queue.cc:260] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:00.687006 10855 ts_tablet_manager.cc:1398] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20240426 09:55:00.688449 9951 tablet_copy_service.cc:342] P f12521b6d51242629cf4297864df00b9: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-1c21d6dedd1441bd977f09cdcf41d436 received from {username='slave'} at 127.0.0.1:40928
I20240426 09:55:00.688745 9951 tablet_copy_service.cc:433] P f12521b6d51242629cf4297864df00b9: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-1c21d6dedd1441bd977f09cdcf41d436 on tablet 1c21d6dedd1441bd977f09cdcf41d436 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:00.855381 10619 raft_consensus.cc:1203] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 09:55:00.871917 10619 raft_consensus.cc:1203] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 09:55:01.036882 10619 raft_consensus.cc:1203] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240426 09:55:01.057152 10619 raft_consensus.cc:1203] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 09:55:01.252573 10816 raft_consensus.cc:1050] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:01.254546 10816 consensus_queue.cc:237] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:01.260342 10619 raft_consensus.cc:1261] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:01.262552 10794 consensus_queue.cc:1035] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:01.264562 10090 raft_consensus.cc:1261] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:01.266887 10014 raft_consensus.cc:1261] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:01.268196 10816 consensus_queue.cc:1035] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 09:55:01.271404 10863 consensus_queue.cc:1035] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:01.290140 10794 raft_consensus.cc:2932] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.294912 10014 raft_consensus.cc:2932] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.304762 10619 raft_consensus.cc:2932] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.306003 10089 raft_consensus.cc:2932] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.311769 9838 catalog_manager.cc:5484] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 reported cstate change: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:01.341184 9936 consensus_queue.cc:237] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:01.345436 10619 raft_consensus.cc:1261] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:55:01.346820 10816 consensus_queue.cc:1035] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:01.349079 10014 raft_consensus.cc:1261] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:55:01.353853 10794 consensus_queue.cc:1035] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:01.355350 10863 raft_consensus.cc:2932] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.356755 10619 raft_consensus.cc:2932] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.362445 10014 raft_consensus.cc:2932] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.365651 9826 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 951e4c48fcb642e58ab79652d3f86261 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:55:01.373768 9838 catalog_manager.cc:5484] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.384155 10077 tablet_service.cc:1514] Processing DeleteTablet for tablet 951e4c48fcb642e58ab79652d3f86261 with delete_type TABLET_DATA_TOMBSTONED (TS d3a480e831d649139ff016fb68aee3ee not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59730
I20240426 09:55:01.392457 10873 tablet_replica.cc:330] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee: stopping tablet replica
I20240426 09:55:01.394002 10873 raft_consensus.cc:2229] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:55:01.395040 10873 raft_consensus.cc:2258] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:55:01.398113 10873 ts_tablet_manager.cc:1875] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:55:01.399693 10794 raft_consensus.cc:1050] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:01.402017 10794 consensus_queue.cc:237] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:01.412519 10873 ts_tablet_manager.cc:1888] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 09:55:01.413074 10873 log.cc:1196] T 951e4c48fcb642e58ab79652d3f86261 P d3a480e831d649139ff016fb68aee3ee: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-2-root/wals/951e4c48fcb642e58ab79652d3f86261
I20240426 09:55:01.413362 10090 raft_consensus.cc:1261] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:01.414175 10014 raft_consensus.cc:1261] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:01.414669 9823 catalog_manager.cc:4830] TS d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): tablet 951e4c48fcb642e58ab79652d3f86261 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:55:01.414250 10619 raft_consensus.cc:1261] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:01.415059 10863 consensus_queue.cc:1035] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:01.416167 10875 consensus_queue.cc:1035] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:01.417241 10794 consensus_queue.cc:1035] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:01.429754 10863 raft_consensus.cc:2932] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.431316 10619 raft_consensus.cc:2932] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.446089 10014 raft_consensus.cc:2932] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.449618 9835 catalog_manager.cc:5484] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.451050 10090 raft_consensus.cc:2932] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.484408 9936 consensus_queue.cc:237] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:01.495218 10875 raft_consensus.cc:1050] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:01.497429 10875 consensus_queue.cc:237] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:01.498062 10619 raft_consensus.cc:1261] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:55:01.498818 10014 raft_consensus.cc:1261] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:55:01.499205 10863 consensus_queue.cc:1035] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:01.500130 10794 consensus_queue.cc:1035] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:01.507861 10794 raft_consensus.cc:2932] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.511268 10620 raft_consensus.cc:2932] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.515611 10010 raft_consensus.cc:2932] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.518014 9826 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 09b98bf14ce848eca1c7ca1840a516b1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:55:01.529055 10619 raft_consensus.cc:1261] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 2 LEARNER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240426 09:55:01.530853 10014 raft_consensus.cc:1261] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240426 09:55:01.530643 10863 consensus_queue.cc:1035] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:01.538641 10875 consensus_queue.cc:1035] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 09:55:01.540081 10090 raft_consensus.cc:1261] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240426 09:55:01.540570 9838 catalog_manager.cc:5484] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.551026 10875 consensus_queue.cc:1035] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:01.555650 10816 raft_consensus.cc:2932] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.560254 10077 tablet_service.cc:1514] Processing DeleteTablet for tablet 09b98bf14ce848eca1c7ca1840a516b1 with delete_type TABLET_DATA_TOMBSTONED (TS d3a480e831d649139ff016fb68aee3ee not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59730
I20240426 09:55:01.561157 10873 tablet_replica.cc:330] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee: stopping tablet replica
I20240426 09:55:01.562165 10873 raft_consensus.cc:2229] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:55:01.562849 10873 raft_consensus.cc:2258] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:55:01.562711 10619 raft_consensus.cc:2932] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.566831 10873 ts_tablet_manager.cc:1875] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:55:01.574046 10014 raft_consensus.cc:2932] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.577669 9836 catalog_manager.cc:5484] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 reported cstate change: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:01.585044 10090 raft_consensus.cc:2932] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.597148 10749 raft_consensus.cc:1050] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:01.599375 10749 consensus_queue.cc:237] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:01.603250 10873 ts_tablet_manager.cc:1888] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 09:55:01.603652 10873 log.cc:1196] T 09b98bf14ce848eca1c7ca1840a516b1 P d3a480e831d649139ff016fb68aee3ee: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-2-root/wals/09b98bf14ce848eca1c7ca1840a516b1
I20240426 09:55:01.616346 10619 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:01.616765 10010 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:01.618711 10799 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:01.619062 9939 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:01.619503 10749 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:01.622493 10749 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:01.633658 9936 consensus_queue.cc:237] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:01.642819 10620 raft_consensus.cc:1261] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240426 09:55:01.644070 10875 consensus_queue.cc:1035] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:01.649257 10089 raft_consensus.cc:1261] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240426 09:55:01.650566 10794 raft_consensus.cc:2932] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.656193 10816 consensus_queue.cc:1035] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:01.655915 10619 raft_consensus.cc:2932] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.663769 10089 raft_consensus.cc:2932] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.671314 9826 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c21d6dedd1441bd977f09cdcf41d436 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:55:01.675660 10836 raft_consensus.cc:2932] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.678136 10011 raft_consensus.cc:2932] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.683781 9823 catalog_manager.cc:4830] TS d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): tablet 09b98bf14ce848eca1c7ca1840a516b1 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:55:01.684672 9939 raft_consensus.cc:2932] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.686342 10620 raft_consensus.cc:2932] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.701730 9839 catalog_manager.cc:5484] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New cstate: current_term: 2 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.721730 10006 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c21d6dedd1441bd977f09cdcf41d436 with delete_type TABLET_DATA_TOMBSTONED (TS 826f03b2fb9c4b9483623b041a1bf80e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34362
I20240426 09:55:01.720191 9836 catalog_manager.cc:5484] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:01.732674 10888 tablet_replica.cc:330] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e: stopping tablet replica
I20240426 09:55:01.740116 10888 raft_consensus.cc:2229] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 09:55:01.740697 10888 raft_consensus.cc:2258] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 09:55:01.743387 10888 ts_tablet_manager.cc:1875] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:55:01.756536 10089 consensus_queue.cc:237] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:01.761255 10619 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:55:01.762233 10014 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:55:01.762576 10836 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:01.763572 10799 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:01.766499 10888 ts_tablet_manager.cc:1888] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240426 09:55:01.766960 10888 log.cc:1196] T 1c21d6dedd1441bd977f09cdcf41d436 P 826f03b2fb9c4b9483623b041a1bf80e: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-1-root/wals/1c21d6dedd1441bd977f09cdcf41d436
I20240426 09:55:01.770262 10749 raft_consensus.cc:2932] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.772562 9825 catalog_manager.cc:4830] TS 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): tablet 1c21d6dedd1441bd977f09cdcf41d436 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:55:01.775426 10619 raft_consensus.cc:2932] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.785177 9823 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 51125fbde06b4a588fb6171dec56a605 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:55:01.785480 9839 catalog_manager.cc:5484] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.791342 10010 raft_consensus.cc:2932] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:01.799914 9925 tablet_service.cc:1514] Processing DeleteTablet for tablet 51125fbde06b4a588fb6171dec56a605 with delete_type TABLET_DATA_TOMBSTONED (TS f12521b6d51242629cf4297864df00b9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47232
I20240426 09:55:01.801548 10889 tablet_replica.cc:330] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9: stopping tablet replica
I20240426 09:55:01.802525 10889 raft_consensus.cc:2229] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:55:01.803189 10889 raft_consensus.cc:2258] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:55:01.807782 10889 ts_tablet_manager.cc:1875] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:55:01.820199 10889 ts_tablet_manager.cc:1888] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 09:55:01.820487 10889 log.cc:1196] T 51125fbde06b4a588fb6171dec56a605 P f12521b6d51242629cf4297864df00b9: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-0-root/wals/51125fbde06b4a588fb6171dec56a605
I20240426 09:55:01.821779 9826 catalog_manager.cc:4830] TS f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): tablet 51125fbde06b4a588fb6171dec56a605 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:55:02.860848 9936 consensus_queue.cc:237] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
W20240426 09:55:02.866614 9905 consensus_peers.cc:485] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6fde852ed1264555af856522b4abc093. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:02.867272 10085 raft_consensus.cc:1261] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:55:02.867664 10010 raft_consensus.cc:1261] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:55:02.868752 10816 consensus_queue.cc:1035] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:02.869511 10794 consensus_queue.cc:1035] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:02.876376 10816 raft_consensus.cc:2932] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:02.879525 10010 raft_consensus.cc:2932] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:02.880367 10085 raft_consensus.cc:2932] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:02.896641 10010 consensus_queue.cc:237] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:02.897498 9835 catalog_manager.cc:5484] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
W20240426 09:55:02.909991 9980 consensus_peers.cc:485] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 007f1451130c4dfcbd86145d56a6f67e. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:02.909986 10085 raft_consensus.cc:1261] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:02.909986 9936 raft_consensus.cc:1261] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:02.911557 10899 consensus_queue.cc:1035] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:02.912431 10815 consensus_queue.cc:1035] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:02.919915 10780 raft_consensus.cc:2932] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:02.926323 10085 raft_consensus.cc:2932] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:02.935081 9935 consensus_queue.cc:237] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:02.943543 10089 raft_consensus.cc:1261] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:55:02.945320 10894 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240426 09:55:02.946893 9905 consensus_peers.cc:485] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21e4aa688b194564bef9b4398f572caf. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:02.948474 10010 raft_consensus.cc:1261] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:55:02.950663 10896 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:02.965906 9835 catalog_manager.cc:5484] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 2 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:02.979557 9939 raft_consensus.cc:2932] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:02.993933 10896 raft_consensus.cc:2932] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:03.000275 10010 raft_consensus.cc:2932] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:03.014570 10085 consensus_queue.cc:237] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:03.017146 10089 raft_consensus.cc:2932] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:03.020380 9839 catalog_manager.cc:5484] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240426 09:55:03.024500 10054 consensus_peers.cc:485] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e197425a8c74512b4dac9d31a3ce0a5. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:03.031005 10010 raft_consensus.cc:1261] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:55:03.031270 9939 raft_consensus.cc:1261] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:55:03.032299 10749 consensus_queue.cc:1035] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:03.033111 10890 consensus_queue.cc:1035] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:03.059348 10909 raft_consensus.cc:2932] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:03.060689 9935 raft_consensus.cc:2932] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:03.073096 10010 raft_consensus.cc:2932] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:03.082963 9835 catalog_manager.cc:5484] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
W20240426 09:55:03.933712 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 256288us
I20240426 09:55:03.981634 10919 ts_tablet_manager.cc:902] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113)
I20240426 09:55:03.983583 10925 ts_tablet_manager.cc:902] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:04.080683 10619 thread.cc:636] tablet-copy [worker] (thread pool) Time spent creating pthread: real 0.656s user 0.034s sys 0.023s
W20240426 09:55:04.081087 10619 thread.cc:602] tablet-copy [worker] (thread pool) Time spent starting thread: real 0.656s user 0.034s sys 0.023s
I20240426 09:55:04.095700 10922 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:04.109298 10928 ts_tablet_manager.cc:902] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113)
I20240426 09:55:04.108608 10922 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:04.112641 10922 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:04.113584 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 70317.9us
I20240426 09:55:04.131114 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:04.150414 10925 tablet_copy_client.cc:308] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.131:34821
I20240426 09:55:04.152025 10100 tablet_copy_service.cc:140] P d3a480e831d649139ff016fb68aee3ee: Received BeginTabletCopySession request for tablet 2e197425a8c74512b4dac9d31a3ce0a5 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:48202)
I20240426 09:55:04.152269 10919 tablet_copy_client.cc:308] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.129:38113
I20240426 09:55:04.152580 10100 tablet_copy_service.cc:161] P d3a480e831d649139ff016fb68aee3ee: Beginning new tablet copy session on tablet 2e197425a8c74512b4dac9d31a3ce0a5 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:48202: session id = 1fee3da01ebc463396397057fdb81a68-2e197425a8c74512b4dac9d31a3ce0a5
I20240426 09:55:04.153863 9951 tablet_copy_service.cc:140] P f12521b6d51242629cf4297864df00b9: Received BeginTabletCopySession request for tablet 6fde852ed1264555af856522b4abc093 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:40928)
I20240426 09:55:04.154434 9951 tablet_copy_service.cc:161] P f12521b6d51242629cf4297864df00b9: Beginning new tablet copy session on tablet 6fde852ed1264555af856522b4abc093 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:40928: session id = 1fee3da01ebc463396397057fdb81a68-6fde852ed1264555af856522b4abc093
I20240426 09:55:04.157048 10939 ts_tablet_manager.cc:902] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:04.158116 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 93545.2us
I20240426 09:55:04.160790 9951 tablet_copy_source_session.cc:205] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:04.161065 10100 tablet_copy_source_session.cc:205] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:04.163453 10919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fde852ed1264555af856522b4abc093. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:04.167212 10925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e197425a8c74512b4dac9d31a3ce0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:04.179620 10925 tablet_copy_client.cc:783] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:04.180171 10925 tablet_copy_client.cc:647] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:04.182216 10919 tablet_copy_client.cc:783] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:04.182732 10919 tablet_copy_client.cc:647] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:04.186604 10925 tablet_copy_client.cc:523] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:04.194340 10925 tablet_bootstrap.cc:492] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:04.194613 10919 tablet_copy_client.cc:523] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:04.203208 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:04.205778 10919 tablet_bootstrap.cc:492] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:04.209143 10939 tablet_copy_client.cc:308] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.130:44411
I20240426 09:55:04.210808 10026 tablet_copy_service.cc:140] P 826f03b2fb9c4b9483623b041a1bf80e: Received BeginTabletCopySession request for tablet 007f1451130c4dfcbd86145d56a6f67e from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:36762)
I20240426 09:55:04.211309 10026 tablet_copy_service.cc:161] P 826f03b2fb9c4b9483623b041a1bf80e: Beginning new tablet copy session on tablet 007f1451130c4dfcbd86145d56a6f67e from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:36762: session id = 1fee3da01ebc463396397057fdb81a68-007f1451130c4dfcbd86145d56a6f67e
I20240426 09:55:04.217823 10026 tablet_copy_source_session.cc:205] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:04.220240 9982 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:04.221297 10815 raft_consensus.cc:2726] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:04.223562 10939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007f1451130c4dfcbd86145d56a6f67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:04.244309 10919 tablet_bootstrap.cc:492] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:04.245486 10928 tablet_copy_client.cc:308] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.129:38113
I20240426 09:55:04.246016 10919 tablet_bootstrap.cc:492] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:04.246728 10919 ts_tablet_manager.cc:1367] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.041s user 0.026s sys 0.003s
I20240426 09:55:04.247814 9951 tablet_copy_service.cc:140] P f12521b6d51242629cf4297864df00b9: Received BeginTabletCopySession request for tablet 21e4aa688b194564bef9b4398f572caf from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:40928)
I20240426 09:55:04.248389 9951 tablet_copy_service.cc:161] P f12521b6d51242629cf4297864df00b9: Beginning new tablet copy session on tablet 21e4aa688b194564bef9b4398f572caf from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:40928: session id = 1fee3da01ebc463396397057fdb81a68-21e4aa688b194564bef9b4398f572caf
I20240426 09:55:04.249312 10919 raft_consensus.cc:348] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:04.249917 10919 raft_consensus.cc:729] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:04.250360 10919 consensus_queue.cc:260] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:04.255564 10939 tablet_copy_client.cc:783] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:04.256062 10939 tablet_copy_client.cc:647] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:04.260001 9951 tablet_copy_source_session.cc:205] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:04.261385 10925 tablet_bootstrap.cc:492] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:04.261925 10939 tablet_copy_client.cc:523] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:04.262097 10925 tablet_bootstrap.cc:492] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:04.262816 10925 ts_tablet_manager.cc:1367] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.069s user 0.013s sys 0.022s
I20240426 09:55:04.263446 10928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e4aa688b194564bef9b4398f572caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:04.265322 10925 raft_consensus.cc:348] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:04.269491 10939 tablet_bootstrap.cc:492] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:04.269737 10919 ts_tablet_manager.cc:1398] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20240426 09:55:04.271665 9951 tablet_copy_service.cc:342] P f12521b6d51242629cf4297864df00b9: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-6fde852ed1264555af856522b4abc093 received from {username='slave'} at 127.0.0.1:40928
I20240426 09:55:04.272042 9951 tablet_copy_service.cc:433] P f12521b6d51242629cf4297864df00b9: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-6fde852ed1264555af856522b4abc093 on tablet 6fde852ed1264555af856522b4abc093 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:04.266675 10925 raft_consensus.cc:729] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:04.273854 10925 consensus_queue.cc:260] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:04.276155 10925 ts_tablet_manager.cc:1398] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240426 09:55:04.278110 10928 tablet_copy_client.cc:783] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:04.278628 10928 tablet_copy_client.cc:647] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:04.281363 10100 tablet_copy_service.cc:342] P d3a480e831d649139ff016fb68aee3ee: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-2e197425a8c74512b4dac9d31a3ce0a5 received from {username='slave'} at 127.0.0.1:48202
I20240426 09:55:04.281764 10100 tablet_copy_service.cc:433] P d3a480e831d649139ff016fb68aee3ee: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-2e197425a8c74512b4dac9d31a3ce0a5 on tablet 2e197425a8c74512b4dac9d31a3ce0a5 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:04.286988 10928 tablet_copy_client.cc:523] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:04.294366 10928 tablet_bootstrap.cc:492] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:04.299672 10939 tablet_bootstrap.cc:492] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:04.300540 10939 tablet_bootstrap.cc:492] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:04.301134 10939 ts_tablet_manager.cc:1367] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.011s
I20240426 09:55:04.303675 10939 raft_consensus.cc:348] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:04.304431 10939 raft_consensus.cc:729] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:04.305065 10939 consensus_queue.cc:260] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:04.307297 10939 ts_tablet_manager.cc:1398] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20240426 09:55:04.310825 10026 tablet_copy_service.cc:342] P 826f03b2fb9c4b9483623b041a1bf80e: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-007f1451130c4dfcbd86145d56a6f67e received from {username='slave'} at 127.0.0.1:36762
I20240426 09:55:04.311185 10026 tablet_copy_service.cc:433] P 826f03b2fb9c4b9483623b041a1bf80e: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-007f1451130c4dfcbd86145d56a6f67e on tablet 007f1451130c4dfcbd86145d56a6f67e with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:04.314205 10928 tablet_bootstrap.cc:492] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:04.315024 10928 tablet_bootstrap.cc:492] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:04.315565 10928 ts_tablet_manager.cc:1367] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.006s
I20240426 09:55:04.318562 10620 raft_consensus.cc:1203] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 09:55:04.319701 10928 raft_consensus.cc:348] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:04.320885 10928 raft_consensus.cc:729] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:04.321498 10928 consensus_queue.cc:260] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:04.323840 10928 ts_tablet_manager.cc:1398] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20240426 09:55:04.329653 9951 tablet_copy_service.cc:342] P f12521b6d51242629cf4297864df00b9: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-21e4aa688b194564bef9b4398f572caf received from {username='slave'} at 127.0.0.1:40928
I20240426 09:55:04.330057 9951 tablet_copy_service.cc:433] P f12521b6d51242629cf4297864df00b9: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-21e4aa688b194564bef9b4398f572caf on tablet 21e4aa688b194564bef9b4398f572caf with peer 1fee3da01ebc463396397057fdb81a68
W20240426 09:55:04.441857 10037 debug-util.cc:398] Leaking SignalData structure 0x7b0800162c00 after lost signal to thread 8761
W20240426 09:55:04.442876 10037 debug-util.cc:398] Leaking SignalData structure 0x7b08001692e0 after lost signal to thread 9873
W20240426 09:55:04.443770 10037 debug-util.cc:398] Leaking SignalData structure 0x7b08001b1900 after lost signal to thread 9965
W20240426 09:55:04.444649 10037 debug-util.cc:398] Leaking SignalData structure 0x7b08002f41a0 after lost signal to thread 10040
W20240426 09:55:04.445433 10037 debug-util.cc:398] Leaking SignalData structure 0x7b08002fe120 after lost signal to thread 10114
W20240426 09:55:04.446259 10037 debug-util.cc:398] Leaking SignalData structure 0x7b08002dc2a0 after lost signal to thread 10644
I20240426 09:55:04.464607 10620 raft_consensus.cc:1203] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240426 09:55:04.686609 10620 raft_consensus.cc:1203] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 09:55:04.697448 10931 raft_consensus.cc:1050] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:04.699251 10931 consensus_queue.cc:237] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:04.704912 10620 raft_consensus.cc:1261] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:04.706087 10914 consensus_queue.cc:1035] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:04.709980 10007 raft_consensus.cc:1261] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:04.711251 9939 raft_consensus.cc:1261] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:04.711159 10914 consensus_queue.cc:1035] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 09:55:04.713307 10914 consensus_queue.cc:1035] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:04.725152 10890 raft_consensus.cc:2932] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:04.726716 10016 raft_consensus.cc:2932] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:04.740031 9939 raft_consensus.cc:2932] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:04.742053 10620 raft_consensus.cc:2932] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:04.754244 9835 catalog_manager.cc:5484] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:04.780735 10620 raft_consensus.cc:1203] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 09:55:04.792335 10085 consensus_queue.cc:237] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:04.799417 8763 debug-util.cc:398] Leaking SignalData structure 0x7b080000e980 after lost signal to thread 10619
I20240426 09:55:04.801052 10620 raft_consensus.cc:1261] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:55:04.801352 10016 raft_consensus.cc:1261] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:55:04.802937 10936 consensus_queue.cc:1035] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:04.804771 10914 consensus_queue.cc:1035] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:04.810997 10890 raft_consensus.cc:2932] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:04.818457 10620 raft_consensus.cc:2932] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:04.825764 9823 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2e197425a8c74512b4dac9d31a3ce0a5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:55:04.826930 10016 raft_consensus.cc:2932] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:04.852003 9839 catalog_manager.cc:5484] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:04.870138 9925 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e197425a8c74512b4dac9d31a3ce0a5 with delete_type TABLET_DATA_TOMBSTONED (TS f12521b6d51242629cf4297864df00b9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47232
I20240426 09:55:04.884171 10962 tablet_replica.cc:330] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9: stopping tablet replica
I20240426 09:55:04.887559 10962 raft_consensus.cc:2229] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:55:04.888335 10962 raft_consensus.cc:2258] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:55:04.891196 10962 ts_tablet_manager.cc:1875] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:55:04.905881 10962 ts_tablet_manager.cc:1888] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 09:55:04.906165 10962 log.cc:1196] T 2e197425a8c74512b4dac9d31a3ce0a5 P f12521b6d51242629cf4297864df00b9: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-0-root/wals/2e197425a8c74512b4dac9d31a3ce0a5
I20240426 09:55:04.907599 9826 catalog_manager.cc:4830] TS f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): tablet 2e197425a8c74512b4dac9d31a3ce0a5 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:55:04.915823 10923 raft_consensus.cc:1050] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:04.918146 10923 consensus_queue.cc:237] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:04.936318 10620 raft_consensus.cc:1261] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 2 LEARNER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240426 09:55:04.936561 10087 raft_consensus.cc:1261] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240426 09:55:04.937798 10923 consensus_queue.cc:1035] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:04.939824 9936 raft_consensus.cc:1261] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240426 09:55:04.940107 10933 consensus_queue.cc:1035] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:04.941054 10923 consensus_queue.cc:1035] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:04.971462 10933 raft_consensus.cc:2932] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:04.973376 9933 raft_consensus.cc:2932] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.000330 9835 catalog_manager.cc:5484] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 reported cstate change: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:04.996490 10620 raft_consensus.cc:2932] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.014997 10085 raft_consensus.cc:2932] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.027559 10007 consensus_queue.cc:237] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:05.033639 10620 raft_consensus.cc:1261] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240426 09:55:05.035038 10933 consensus_queue.cc:1035] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:05.043264 10972 raft_consensus.cc:2932] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.046394 10085 raft_consensus.cc:1261] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240426 09:55:05.055752 10619 raft_consensus.cc:2932] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.071710 9839 catalog_manager.cc:5484] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New cstate: current_term: 2 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.078130 10970 consensus_queue.cc:1035] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 09:55:05.087548 9825 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 007f1451130c4dfcbd86145d56a6f67e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:55:05.095345 9925 tablet_service.cc:1514] Processing DeleteTablet for tablet 007f1451130c4dfcbd86145d56a6f67e with delete_type TABLET_DATA_TOMBSTONED (TS f12521b6d51242629cf4297864df00b9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47232
I20240426 09:55:05.096520 10962 tablet_replica.cc:330] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9: stopping tablet replica
I20240426 09:55:05.097524 10962 raft_consensus.cc:2229] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 09:55:05.098147 10962 raft_consensus.cc:2258] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 09:55:05.095052 10085 raft_consensus.cc:2932] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.109325 10962 ts_tablet_manager.cc:1875] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:55:05.143898 10962 ts_tablet_manager.cc:1888] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240426 09:55:05.144359 10962 log.cc:1196] T 007f1451130c4dfcbd86145d56a6f67e P f12521b6d51242629cf4297864df00b9: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-0-root/wals/007f1451130c4dfcbd86145d56a6f67e
I20240426 09:55:05.150970 10934 raft_consensus.cc:1050] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:05.152889 10934 consensus_queue.cc:237] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:05.165226 10918 raft_consensus.cc:1050] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:05.166268 9826 catalog_manager.cc:4830] TS f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): tablet 007f1451130c4dfcbd86145d56a6f67e (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:55:05.167491 10918 consensus_queue.cc:237] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:05.177215 10619 raft_consensus.cc:1261] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:05.177603 10620 raft_consensus.cc:1261] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:05.182333 10918 consensus_queue.cc:1035] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:05.184484 10918 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:05.185474 10085 raft_consensus.cc:1261] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:05.185793 10016 raft_consensus.cc:1261] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:05.187741 10816 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:05.188544 10894 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:05.190379 10085 raft_consensus.cc:1261] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:05.191653 10934 consensus_queue.cc:1035] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:05.198668 10007 raft_consensus.cc:1261] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:05.204841 10894 consensus_queue.cc:1035] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:05.256206 10975 raft_consensus.cc:2932] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.262090 10085 raft_consensus.cc:2932] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.260767 10918 raft_consensus.cc:2932] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.269812 10087 raft_consensus.cc:2932] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.266160 10007 raft_consensus.cc:2932] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.274235 10016 raft_consensus.cc:2932] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.260303 10620 raft_consensus.cc:2932] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.283195 9835 catalog_manager.cc:5484] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 reported cstate change: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:05.300580 9839 catalog_manager.cc:5484] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.315896 10620 raft_consensus.cc:2932] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.322662 9936 consensus_queue.cc:237] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:05.332890 10007 raft_consensus.cc:1261] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:55:05.334756 10982 consensus_queue.cc:1035] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:05.340351 10620 raft_consensus.cc:1261] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:55:05.343909 10983 consensus_queue.cc:1035] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 09:55:05.347720 10983 raft_consensus.cc:2932] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.353819 10007 raft_consensus.cc:2932] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.354805 9933 consensus_queue.cc:237] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:05.362068 10619 raft_consensus.cc:1261] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:55:05.363444 10975 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:05.363834 10087 raft_consensus.cc:1261] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:55:05.366310 10982 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:05.382882 10918 raft_consensus.cc:2932] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.389577 9826 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6fde852ed1264555af856522b4abc093 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:55:05.390108 10619 raft_consensus.cc:2932] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.372463 10620 raft_consensus.cc:2932] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.415939 9838 catalog_manager.cc:5484] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.419642 10087 raft_consensus.cc:2932] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.454746 10077 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fde852ed1264555af856522b4abc093 with delete_type TABLET_DATA_TOMBSTONED (TS d3a480e831d649139ff016fb68aee3ee not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59730
I20240426 09:55:05.465663 9838 catalog_manager.cc:5484] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index 3 to 4, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New cstate: current_term: 1 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:05.474043 9826 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21e4aa688b194564bef9b4398f572caf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:55:05.538141 10984 tablet_replica.cc:330] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee: stopping tablet replica
I20240426 09:55:05.539273 10984 raft_consensus.cc:2229] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:55:05.540045 10984 raft_consensus.cc:2258] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:55:05.550868 10984 ts_tablet_manager.cc:1875] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:55:05.664403 10984 ts_tablet_manager.cc:1888] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 09:55:05.664726 10984 log.cc:1196] T 6fde852ed1264555af856522b4abc093 P d3a480e831d649139ff016fb68aee3ee: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-2-root/wals/6fde852ed1264555af856522b4abc093
I20240426 09:55:06.599113 10972 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:06.599822 10972 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.601917 10972 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.619450 10986 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:06.658550 9870 debug-util.cc:398] Leaking SignalData structure 0x7b080003cc80 after lost signal to thread 8761
I20240426 09:55:06.685537 10985 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:06.761407 9870 debug-util.cc:398] Leaking SignalData structure 0x7b0800036360 after lost signal to thread 9873
I20240426 09:55:06.762089 10986 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.764269 10986 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.690296 10987 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:06.765337 10987 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.729059 10989 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:06.739953 10970 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:06.766400 10989 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:06.766522 10970 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:06.767642 10987 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.768672 10970 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:06.769112 10989 leader_election.cc:288] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.769649 10989 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:06.770097 10989 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:06.742208 10994 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:06.744808 10971 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:06.749820 10992 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:06.798596 10992 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.761015 10985 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.802712 11003 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:06.686187 10951 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:06.803216 11003 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.803364 10951 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:06.805207 11003 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.805384 10951 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:06.806394 11003 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:06.760089 9838 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50254 (request call id 187) took 1297 ms. Trace:
I20240426 09:55:06.807958 10998 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:06.805933 10951 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:06.808655 10998 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.809000 10951 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:06.810108 10986 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:06.811645 9870 debug-util.cc:398] Leaking SignalData structure 0x7b08001388e0 after lost signal to thread 9965
W20240426 09:55:06.812983 9870 debug-util.cc:398] Leaking SignalData structure 0x7b08002c2e40 after lost signal to thread 9966
I20240426 09:55:06.810629 10986 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.799113 10989 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.801699 11001 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:06.813089 10970 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:06.810895 10972 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:06.812222 10998 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.890250 10989 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:06.892319 10989 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:06.806914 9838 rpcz_store.cc:276] 0426 09:55:05.462653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:05.462800 (+ 147us) service_pool.cc:225] Handling call
0426 09:55:05.486687 (+ 23887us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:05.486713 (+ 26us) write_op.cc:446] Released schema lock
0426 09:55:06.760002 (+1273289us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:05.470827 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:05.471274 (+ 447us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:05.471308 (+ 34us) write_op.cc:427] Acquired schema lock
0426 09:55:05.471323 (+ 15us) tablet.cc:660] Decoding operations
0426 09:55:05.471600 (+ 277us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:05.471645 (+ 45us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:05.471670 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0426 09:55:05.471826 (+ 156us) tablet.cc:699] Row locks acquired
0426 09:55:05.471842 (+ 16us) write_op.cc:260] PREPARE: finished
0426 09:55:05.471944 (+ 102us) write_op.cc:270] Start()
0426 09:55:05.472027 (+ 83us) write_op.cc:276] Timestamp: P: 1714125305471909 usec, L: 0
0426 09:55:05.472044 (+ 17us) op_driver.cc:331] REPLICATION: starting
0426 09:55:05.472502 (+ 458us) log.cc:842] Serialized 535 byte log entry
0426 09:55:05.474919 (+ 2417us) op_driver.cc:444] REPLICATION: finished
0426 09:55:05.475294 (+ 375us) write_op.cc:301] APPLY: starting
0426 09:55:05.475355 (+ 61us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:05.475443 (+ 88us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:05.475543 (+ 100us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:05.475694 (+ 151us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:05.475819 (+ 125us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0426 09:55:05.475861 (+ 42us) write_op.cc:312] APPLY: finished
0426 09:55:05.476014 (+ 153us) log.cc:842] Serialized 22 byte log entry
0426 09:55:05.476144 (+ 130us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:05.476207 (+ 63us) write_op.cc:446] Released schema lock
0426 09:55:05.476238 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":200,"apply.run_cpu_time_us":1162,"apply.run_wall_time_us":1161,"num_ops":1,"prepare.queue_time_us":88,"prepare.run_cpu_time_us":1899,"prepare.run_wall_time_us":1899,"replication_time_us":2792}]]}
I20240426 09:55:06.817656 10982 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:06.887763 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:06.873895 10987 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:06.906622 10995 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:06.903848 10006 tablet_service.cc:1514] Processing DeleteTablet for tablet 21e4aa688b194564bef9b4398f572caf with delete_type TABLET_DATA_TOMBSTONED (TS 826f03b2fb9c4b9483623b041a1bf80e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34362
I20240426 09:55:06.887998 10951 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.811601 10971 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.911520 10993 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:06.882002 11003 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.901604 10989 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.915647 10997 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:06.919297 10985 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.889672 10972 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.886746 10970 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.811033 10994 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.923740 10993 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:06.916849 10951 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:06.925997 10990 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:06.886865 11001 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:06.908061 10987 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.903388 10982 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.922732 10989 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:06.907142 10995 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:06.956303 10975 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:06.956797 10995 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.957145 10936 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:06.957307 10996 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:06.928692 11000 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:06.957665 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:06.961383 10620 raft_consensus.cc:2454] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:06.957628 10936 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.957809 10996 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.959115 10992 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.973461 10970 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.974858 10996 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.974961 10970 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:06.975375 11006 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:06.975456 10970 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.934489 10951 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:06.975947 11006 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:06.964722 10986 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.979395 10971 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.986696 10951 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.987039 10971 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:06.987175 10951 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:06.987500 10971 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.987629 10951 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:06.989161 9836 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50240 (request call id 196) took 1521 ms. Trace:
I20240426 09:55:06.989310 10971 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.989848 10971 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
W20240426 09:55:06.973975 9870 debug-util.cc:398] Leaking SignalData structure 0x7b080032e220 after lost signal to thread 10040
I20240426 09:55:06.990118 10951 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.981546 10995 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
W20240426 09:55:06.991070 9870 debug-util.cc:398] Leaking SignalData structure 0x7b080031f520 after lost signal to thread 10114
I20240426 09:55:06.990937 10995 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:06.928970 10997 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:06.986539 10972 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.973449 10993 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:55:06.991390 9870 debug-util.cc:398] Leaking SignalData structure 0x7b080031f540 after lost signal to thread 10115
I20240426 09:55:06.980115 11003 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.956799 10975 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.977970 10970 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.955315 10989 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:06.994199 9839 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58806 (request call id 80) took 1450 ms. Trace:
I20240426 09:55:07.008774 9839 rpcz_store.cc:276] 0426 09:55:05.544209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:05.544528 (+ 319us) service_pool.cc:225] Handling call
0426 09:55:06.994140 (+1449612us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1222838,"reported_tablets":1}
I20240426 09:55:06.989547 9836 rpcz_store.cc:276] 0426 09:55:05.467830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:05.467930 (+ 100us) service_pool.cc:225] Handling call
0426 09:55:06.989096 (+1521166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":545,"reported_tablets":2}
I20240426 09:55:06.971894 11001 leader_election.cc:288] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:07.007527 10991 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.007892 10995 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:06.979405 11006 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:06.959435 10990 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.012328 10997 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:06.989666 10982 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.013540 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:06.996243 10931 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.015077 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:06.969256 11000 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:06.997779 10972 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:07.019991 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:07.020671 10090 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 1.
I20240426 09:55:07.022607 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:07.022909 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:07.020828 10999 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.021106 10988 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:55:07.024119 9870 debug-util.cc:398] Leaking SignalData structure 0x7b0800328ce0 after lost signal to thread 10644
I20240426 09:55:07.023309 10086 raft_consensus.cc:2454] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 1.
I20240426 09:55:06.998019 10971 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:07.024726 9870 debug-util.cc:398] Leaking SignalData structure 0x7b0800328d00 after lost signal to thread 10645
I20240426 09:55:07.024641 10987 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.029917 10987 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:07.030331 10987 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.032052 10987 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.024437 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:07.034664 10088 raft_consensus.cc:2454] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:07.036193 10989 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:07.039122 11005 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.021134 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:07.020860 10993 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:07.039055 11001 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:07.048321 10990 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.049829 11001 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.050278 10994 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.051556 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:07.052150 10088 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 1.
I20240426 09:55:07.040966 10975 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.049855 11004 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.024356 10999 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.028091 10972 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.048852 10936 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:07.016845 10991 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.039614 11005 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.019503 10931 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.051095 10087 raft_consensus.cc:2454] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 1.
I20240426 09:55:07.040359 10988 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:07.072371 10089 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 1.
I20240426 09:55:07.072643 10972 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.053206 10993 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:07.074024 10988 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:07.076656 11000 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.057616 11004 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.077152 10993 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:07.079819 11004 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:07.080516 10583 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:07.081360 10989 raft_consensus.cc:2781] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:07.081702 10989 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:07.081760 10583 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:07.082152 10989 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:07.083607 10999 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:07.084975 10931 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:07.101514 10997 raft_consensus.cc:2781] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:07.101919 10997 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.102278 10997 raft_consensus.cc:3037] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:07.097021 11001 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:07.117004 10971 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.115743 10583 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:07.134117 11005 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:07.128412 10995 raft_consensus.cc:2781] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:07.135922 11001 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:07.133275 10975 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.132457 10583 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:07.136291 10995 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:07.138943 10991 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:07.143641 10994 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:07.150655 10998 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.143608 10990 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:07.154462 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 271435us
I20240426 09:55:07.155674 11010 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:07.155958 10982 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.156370 10996 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:07.156713 10982 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.156865 10996 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.147243 10995 raft_consensus.cc:3037] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:07.158433 9980 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:07.162920 10583 leader_election.cc:302] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:07.163403 10996 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.148636 10999 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:07.167163 11017 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.143955 11001 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.152161 10951 raft_consensus.cc:2781] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:07.152113 10918 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:07.156230 10894 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:07.144999 10975 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.152079 11004 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.155601 10989 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:07.173645 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.175040 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.176805 10918 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.163791 10583 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:07.178617 10975 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.171205 10951 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:07.182396 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.183238 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.250988 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:07.164085 11005 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.280652 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.281301 9940 raft_consensus.cc:2454] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 1.
I20240426 09:55:07.173043 10993 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:07.283486 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:07.164512 10990 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.167698 10998 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.161739 10994 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:07.171892 10999 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.156265 11010 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.285319 10982 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.494828 11018 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.494882 11019 tablet_replica.cc:330] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e: stopping tablet replica
I20240426 09:55:07.317528 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.313148 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:07.505693 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:07.369982 10988 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.476135 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:07.494012 11013 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:07.254616 10951 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:07.505121 9933 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
W20240426 09:55:07.285235 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 126865us
I20240426 09:55:07.472703 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:07.508895 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.342811 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.493682 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:07.248143 11002 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:07.183753 10996 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:07.183468 10894 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.472407 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:07.378844 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:07.569087 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.768464 10990 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:07.760972 10995 consensus_meta.cc:287] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68: Time spent flushing consensus metadata: real 0.593s user 0.006s sys 0.001s
I20240426 09:55:07.724897 11024 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.317121 10975 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:07.748391 11028 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:07.735678 10997 consensus_meta.cc:287] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68: Time spent flushing consensus metadata: real 0.619s user 0.009s sys 0.000s
I20240426 09:55:07.696239 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.503619 11018 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.642977 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.734081 10951 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:07.253986 10992 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.249272 10931 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.772089 9935 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:07.568687 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.256646 11004 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.506750 10988 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.798858 10931 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.768764 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.463213 10970 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:07.412042 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:07.346671 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:07.781963 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:07.548483 11020 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:07.769759 10995 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:07.687723 11009 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:07.256008 10991 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.503532 10989 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.254858 10936 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:07.306126 10993 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.770350 10975 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:07.773032 10894 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.802701 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:07.804116 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:07.804736 10010 raft_consensus.cc:2454] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 1.
I20240426 09:55:07.312072 11003 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:07.373013 10986 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:07.548328 11011 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:07.773682 10997 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:07.804167 10016 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 1.
I20240426 09:55:07.593616 11001 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:07.513721 10918 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.469898 11008 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.490231 11016 raft_consensus.cc:2781] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:07.753715 10983 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.644794 11000 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:07.626202 10985 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:07.732566 10998 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.316540 11014 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:07.625123 11006 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:07.836975 10090 raft_consensus.cc:2454] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:07.336654 10586 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:07.832926 10970 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.853673 10986 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.379146 10987 raft_consensus.cc:2781] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:07.282838 11005 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.843099 10936 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.863695 10983 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.864367 10986 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.864892 11005 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:07.796833 10990 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:07.865053 10936 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:07.865314 10970 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.865830 10987 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:07.792817 10992 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:07.865695 11006 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.354265 11022 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.866503 10987 raft_consensus.cc:3037] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:07.803822 10996 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.865463 10990 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.260610 11017 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.846201 10989 raft_consensus.cc:2781] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:07.854187 10994 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:07.871814 11019 raft_consensus.cc:2229] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:55:07.868806 10992 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:07.846089 11003 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.837255 10991 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:07.860826 11000 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.873170 11019 raft_consensus.cc:2258] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:55:07.507321 11013 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.800076 11002 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.869988 11006 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.772631 11024 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:07.867307 11008 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.868553 11011 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.862694 11016 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.801735 11028 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.866566 11022 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.859899 11020 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.883901 11030 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.885828 11027 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:07.799860 9933 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:07.907554 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.766959 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.909468 9933 raft_consensus.cc:2454] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:07.909900 9932 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:07.351416 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:07.911717 9979 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.026s sys 0.034s
W20240426 09:55:07.912326 9979 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.565s user 0.026s sys 0.034s
I20240426 09:55:07.801173 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.782155 9938 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:07.915627 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.571050 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.916543 9940 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:07.915351 9937 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:07.914634 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:07.918995 9936 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:07.913607 9935 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:07.831250 10008 raft_consensus.cc:2454] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 1.
I20240426 09:55:07.928987 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:07.929821 10011 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
W20240426 09:55:07.845724 10056 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.018s sys 0.066s
W20240426 09:55:07.947381 10056 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.698s user 0.019s sys 0.067s
I20240426 09:55:07.951274 9823 catalog_manager.cc:4830] TS d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): tablet 6fde852ed1264555af856522b4abc093 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:55:07.959914 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:07.962631 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:07.964572 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:07.964681 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:07.965417 10086 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:07.845475 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:07.802590 10583 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.031s sys 0.045s
W20240426 09:55:07.970822 10583 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.687s user 0.041s sys 0.047s
W20240426 09:55:07.971386 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 807633us
I20240426 09:55:07.491048 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
W20240426 09:55:07.860285 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 577995us
I20240426 09:55:07.980917 10951 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.981819 10951 raft_consensus.cc:2726] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:07.865528 10983 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:07.982023 10991 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:07.982127 10975 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:07.982901 10991 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.983107 10988 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:07.871050 10989 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.983866 10988 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:07.984031 10989 raft_consensus.cc:3037] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:07.984549 10988 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.860675 10985 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.987280 10988 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:07.987804 10988 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.988508 10988 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.989881 10931 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:07.990512 10931 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:07.990343 10991 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.991189 10931 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:07.992880 10619 raft_consensus.cc:1226] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:08.000829 10936 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.001188 10995 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.002050 10972 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:08.002892 10972 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.003947 10936 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.005265 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.014825 11002 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:07.880079 11016 raft_consensus.cc:3037] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.036391 11022 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.039688 11024 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.039960 11015 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:08.029474 10931 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.045446 11023 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:08.051060 11018 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.034626 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.058729 10991 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.061146 11004 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.061762 11008 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.062515 10619 raft_consensus.cc:1226] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:08.027400 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:07.884737 11030 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.046319 11023 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.036517 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:08.074059 9934 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:08.074360 11028 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.076115 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:07.875670 11014 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.078737 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.079123 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:08.079547 10997 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.079607 10087 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:07.886266 11027 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:07.857273 11009 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.034547 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:08.015702 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:08.082909 10988 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.093662 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
W20240426 09:55:08.092903 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 115532us
I20240426 09:55:08.084527 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.087139 10996 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.100683 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:08.101002 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.101320 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.102825 10090 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.102169 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.106017 11017 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.193145 11033 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.193449 10583 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:08.106148 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:08.186729 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 79016.1us
I20240426 09:55:08.158047 10619 raft_consensus.cc:1799] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Handling vote request from an unknown peer 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:08.196527 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.196858 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.102233 10615 raft_consensus.cc:2454] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:08.203606 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.205966 10619 raft_consensus.cc:2379] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:08.210201 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:08.099525 11015 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.101802 11010 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.101547 11020 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.098222 10999 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.149857 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:08.150161 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:08.219262 11015 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.114682 11036 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.223985 11023 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.224093 10972 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:08.224560 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 120811us
I20240426 09:55:08.189767 11011 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.151942 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.190013 10993 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.115818 10936 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.192135 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.227561 10586 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:55:08.228412 9982 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.141.129:38113 timed out after 1.538s (SENT) [suppressed 1 similar messages]
I20240426 09:55:08.111025 10613 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:08.190101 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:08.235103 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.235317 10586 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:08.210755 10989 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.129923 11032 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:08.205533 11019 ts_tablet_manager.cc:1875] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:55:08.238816 11003 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.239460 10997 raft_consensus.cc:2781] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.239624 10990 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.239799 11024 raft_consensus.cc:2726] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:08.240680 10612 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:08.224372 11036 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.244318 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.194023 11033 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.250219 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.250455 11027 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.252874 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:08.251991 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.255280 10617 raft_consensus.cc:2379] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 2: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:08.257098 9933 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 1.
I20240426 09:55:08.247031 10997 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:08.215117 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:08.261804 10054 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:08.264393 11013 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.264235 10990 consensus_queue.cc:1046] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 09:55:08.262986 10997 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.263748 10015 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:08.247745 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.266602 10090 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:08.266170 11003 raft_consensus.cc:3032] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:08.267516 11003 raft_consensus.cc:729] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:08.268664 11003 consensus_queue.cc:260] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.270071 11003 raft_consensus.cc:3037] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:08.263094 10087 raft_consensus.cc:3037] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.262936 11009 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.273545 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.273888 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.277755 11004 raft_consensus.cc:2781] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:08.279156 11004 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:08.279573 11004 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:08.273479 10616 raft_consensus.cc:1226] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:08.281692 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.283911 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.279942 10586 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:08.287866 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.287119 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.291000 11037 raft_consensus.cc:2781] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:08.296813 10583 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:08.297312 11014 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.280869 11030 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.286913 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.291787 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.280773 11016 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.290969 9905 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:08.266662 11010 consensus_queue.cc:1046] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20240426 09:55:08.306001 11024 raft_consensus.cc:2781] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.307871 11024 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:08.309995 11024 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.311264 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.312875 11014 raft_consensus.cc:2781] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:08.306651 11033 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.313495 11014 raft_consensus.cc:686] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:08.313791 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.313688 11033 consensus_queue.cc:1046] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20240426 09:55:08.317154 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.317286 10620 raft_consensus.cc:2379] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 2: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:08.317248 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.319381 10089 raft_consensus.cc:2454] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:08.306640 10989 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.278108 11032 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.324175 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.333214 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:08.333729 11016 leader_election.cc:288] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.335667 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.336035 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.324695 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.324819 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.328213 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.330924 10985 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.342463 10081 raft_consensus.cc:1799] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Handling vote request from an unknown peer 826f03b2fb9c4b9483623b041a1bf80e
W20240426 09:55:08.342808 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 54914.9us
I20240426 09:55:08.339319 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.339725 10087 raft_consensus.cc:2454] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:08.340283 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.331108 11000 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.342105 10007 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:08.318341 11014 consensus_queue.cc:237] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.323060 10614 raft_consensus.cc:1226] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:08.344466 10086 raft_consensus.cc:1226] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:08.345636 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:08.352186 10088 raft_consensus.cc:3037] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.348906 10012 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:08.346422 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.347551 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:08.355347 10970 raft_consensus.cc:3032] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:08.355777 10970 raft_consensus.cc:729] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:08.349292 10619 raft_consensus.cc:1226] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:08.343586 10996 raft_consensus.cc:2781] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.350433 10583 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:08.352353 10016 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:08.359114 10081 raft_consensus.cc:2379] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:08.359540 10989 raft_consensus.cc:2781] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:08.360087 10989 raft_consensus.cc:686] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:08.361065 10989 consensus_queue.cc:237] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.357565 10996 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:08.363370 10996 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.359709 10970 consensus_queue.cc:260] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.365621 11004 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.366649 10081 raft_consensus.cc:1226] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:08.356416 9905 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:08.372942 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:08.374159 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 145979us
I20240426 09:55:08.376773 10970 raft_consensus.cc:3037] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.349383 10987 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.384884 10992 raft_consensus.cc:2781] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.385318 10992 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:08.385706 10992 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.376055 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.388517 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.389204 10087 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:08.390301 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.389937 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:08.395886 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.396917 11000 consensus_queue.cc:1046] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:08.398075 11000 raft_consensus.cc:3032] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:08.398425 11000 raft_consensus.cc:729] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:08.391837 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.391163 11004 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:08.400650 10081 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:08.388286 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.390739 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.402535 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 695) took 1152 ms. Trace:
I20240426 09:55:08.402884 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.395532 11036 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.407675 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.416347 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.416707 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.416065 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.410194 10085 rpcz_store.cc:276] 0426 09:55:07.250082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:07.252162 (+ 2080us) service_pool.cc:225] Handling call
0426 09:55:07.252432 (+ 270us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:08.015584 (+763152us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:08.015628 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:08.015676 (+ 48us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:08.015705 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:08.402158 (+386453us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b54001077d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:08.402190 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:08.402360 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1754109184}
I20240426 09:55:08.407992 11000 consensus_queue.cc:260] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.420570 10007 raft_consensus.cc:2454] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:08.422118 10015 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:08.422268 11000 raft_consensus.cc:3037] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.420519 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.420424 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.425966 10620 raft_consensus.cc:1226] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:08.415732 10008 raft_consensus.cc:1226] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:08.431829 10088 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:08.417860 10012 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:08.433246 10086 raft_consensus.cc:1226] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:08.435966 10583 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:08.436661 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.437227 10011 raft_consensus.cc:2454] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:08.440061 10990 raft_consensus.cc:3032] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:08.440459 10990 raft_consensus.cc:729] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:08.442281 10989 raft_consensus.cc:2781] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:08.443017 10989 raft_consensus.cc:686] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:08.442668 10990 consensus_queue.cc:260] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.444183 10989 consensus_queue.cc:237] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.444267 10990 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.446943 10987 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:08.451560 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:08.458385 11024 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.460260 11024 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.461519 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:08.462915 11037 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:08.463486 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.450305 10970 consensus_queue.cc:1046] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20240426 09:55:08.465795 10997 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.466316 10987 raft_consensus.cc:3032] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:08.466733 10987 raft_consensus.cc:729] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:08.467087 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.466506 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.467628 10987 consensus_queue.cc:260] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.468633 10987 raft_consensus.cc:3037] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.463339 11037 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:08.472620 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:08.473266 9940 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.465222 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.475898 10585 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:08.477195 10585 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:08.477924 11014 raft_consensus.cc:2781] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.478279 11014 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.478699 11014 raft_consensus.cc:3037] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.478866 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.479682 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.485230 9940 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:08.485967 10586 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:08.478976 11024 raft_consensus.cc:2781] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.487174 11016 raft_consensus.cc:2781] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:08.487239 11024 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:08.487725 11016 raft_consensus.cc:686] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:08.488322 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:08.488651 11016 consensus_queue.cc:237] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.487742 11024 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.487893 11032 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:08.490749 10615 raft_consensus.cc:1226] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:08.492785 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.483472 10990 consensus_queue.cc:1046] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20240426 09:55:08.494057 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.496063 10990 consensus_queue.cc:1225] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer 1fee3da01ebc463396397057fdb81a68 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240426 09:55:08.496484 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.499841 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.500254 10055 leader_election.cc:302] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:08.501118 10999 raft_consensus.cc:3037] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.450735 10996 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.504765 11010 raft_consensus.cc:3032] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:08.505026 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.505203 11010 raft_consensus.cc:729] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:08.506306 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.506024 11010 consensus_queue.cc:260] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.478329 10585 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:08.507284 11010 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.508658 11032 consensus_queue.cc:1046] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:08.509744 11032 raft_consensus.cc:3032] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:08.511360 10997 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.512224 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:08.512913 10992 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.510325 11032 raft_consensus.cc:729] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:08.514847 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.514479 11032 consensus_queue.cc:260] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.516937 10992 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:08.517817 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:08.516551 11032 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.518347 10989 raft_consensus.cc:2781] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.522217 9904 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:08.522513 10989 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.522986 10989 raft_consensus.cc:3037] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.523963 11014 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.524590 10999 raft_consensus.cc:2726] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:08.528333 10996 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:08.529196 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:08.529373 9904 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:08.529963 10615 raft_consensus.cc:2379] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 2: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:08.530174 10992 raft_consensus.cc:2781] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.530566 10992 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:08.530954 10992 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.533026 10990 raft_consensus.cc:2781] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.533421 10990 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:08.533797 10990 raft_consensus.cc:3037] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.536137 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.537299 10055 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:08.538362 10055 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:08.539209 11001 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.539567 11001 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.539899 11001 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.540825 10936 raft_consensus.cc:2726] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:08.545349 10055 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:08.547839 10989 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.547681 10936 raft_consensus.cc:2781] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.548817 10936 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.549155 10936 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.509429 11024 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.552767 11001 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.555037 11001 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.561422 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.559597 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.566800 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.568737 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.569813 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:08.565255 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.573719 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.574391 10086 raft_consensus.cc:1226] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:08.577721 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.577749 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.563881 10990 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.582880 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.583398 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.584622 10615 raft_consensus.cc:1226] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:08.586159 11004 consensus_queue.cc:1046] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:08.586355 10585 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:08.590433 10585 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:08.591027 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:08.588999 10010 raft_consensus.cc:2454] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:08.576061 10989 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.593940 10055 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:08.594861 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.595083 10997 raft_consensus.cc:2726] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:08.599622 9941 raft_consensus.cc:2454] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:55:08.594738 11016 raft_consensus.cc:2726] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:08.600494 11038 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.600886 11038 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.601212 11038 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.591750 11004 consensus_queue.cc:1225] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer 1fee3da01ebc463396397057fdb81a68 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240426 09:55:08.605283 10586 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:08.606165 11037 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.611011 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.611164 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.611924 11011 raft_consensus.cc:3032] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:08.612294 11011 raft_consensus.cc:729] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:08.612854 11037 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.613765 10615 raft_consensus.cc:1226] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:08.613111 11011 consensus_queue.cc:260] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.615520 10987 consensus_queue.cc:1046] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:08.616498 11011 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.616569 10987 raft_consensus.cc:3032] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:08.617031 10987 raft_consensus.cc:729] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:08.617772 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.603739 11014 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.619091 10055 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:08.605074 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.620131 10987 consensus_queue.cc:260] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.621752 10012 raft_consensus.cc:2454] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:08.622553 10011 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:08.623337 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:08.624871 11043 log.cc:924] Time spent T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20240426 09:55:08.613251 10585 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:08.621418 10987 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:08.628732 10990 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.608096 11024 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.629523 11024 raft_consensus.cc:2726] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:08.630177 10585 leader_election.cc:302] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:08.630460 10992 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.632380 11014 raft_consensus.cc:2781] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:08.632840 11014 raft_consensus.cc:686] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:08.632946 10055 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:08.633515 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.623286 9836 catalog_manager.cc:5484] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 1 to 2, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 2 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:08.634526 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.634135 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.635177 9836 catalog_manager.cc:5484] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 1 to 2, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 2 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:08.636950 10585 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:08.637689 11037 raft_consensus.cc:2726] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:08.639303 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.639629 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.640280 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.640645 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.641103 10999 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.641479 10999 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.641736 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.641825 10999 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.642647 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.642885 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.643100 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.636853 9836 catalog_manager.cc:5484] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 1 to 2, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 2 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:08.641125 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.645059 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.647967 9982 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:08.609828 11016 raft_consensus.cc:2781] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:08.641824 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.650395 11001 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.652699 11016 raft_consensus.cc:686] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:08.654047 10013 raft_consensus.cc:2454] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:08.653493 11001 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.654477 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.653571 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.654990 11001 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.643952 11014 consensus_queue.cc:237] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.658082 11038 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.655719 10936 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.659364 10082 raft_consensus.cc:1226] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:08.659965 11038 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.623745 11019 ts_tablet_manager.cc:1888] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 09:55:08.660600 11032 raft_consensus.cc:2726] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:08.661033 11019 log.cc:1196] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-1-root/wals/21e4aa688b194564bef9b4398f572caf
I20240426 09:55:08.663262 10999 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.657269 10992 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.662012 10990 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:08.664515 10990 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.665436 10999 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.666471 10990 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.644712 9836 catalog_manager.cc:5484] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 1 to 2, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 2 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:08.668510 10082 consensus_queue.cc:237] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:08.673780 10936 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.674319 10585 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:08.674736 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.663645 11019 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34362 (request call id 693) took 1780 ms. Trace:
I20240426 09:55:08.676159 11037 raft_consensus.cc:2781] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:08.676640 11037 raft_consensus.cc:686] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:08.677155 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.640493 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:08.664772 10055 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:08.680227 10615 raft_consensus.cc:1226] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:08.676477 10992 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:08.677532 11037 consensus_queue.cc:237] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.680713 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.683939 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.685114 11038 raft_consensus.cc:2781] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.680506 11016 consensus_queue.cc:237] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.695420 10055 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:08.704001 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.704578 10088 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:08.676201 10990 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:08.707614 10055 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:08.708645 9982 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:08.714079 10012 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:55:08.714706 11011 raft_consensus.cc:2726] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:08.707125 10992 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.721303 10992 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.721001 10055 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:08.708549 10936 raft_consensus.cc:2781] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.695178 11001 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.722610 10936 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.723057 10936 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.723807 11001 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.707264 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.726085 10585 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:08.703858 11003 consensus_queue.cc:1046] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20240426 09:55:08.726881 11014 raft_consensus.cc:2781] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:08.695190 11038 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.731854 9982 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:08.732435 10055 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:08.727334 11014 raft_consensus.cc:686] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 3 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:08.728101 11038 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.733322 10972 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:08.734587 11001 raft_consensus.cc:2781] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.734968 10972 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:08.735085 11001 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.736560 10987 raft_consensus.cc:2781] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.736946 10987 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:08.735472 9982 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:08.739909 10988 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.740293 10988 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.740680 10988 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.735437 10972 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:08.737989 10987 raft_consensus.cc:3037] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.744503 10992 raft_consensus.cc:3032] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:08.675573 11019 rpcz_store.cc:276] 0426 09:55:06.882754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:06.886889 (+ 4135us) service_pool.cc:225] Handling call
0426 09:55:07.171579 (+284690us) ts_tablet_manager.cc:1120] Deleting tablet 21e4aa688b194564bef9b4398f572caf
0426 09:55:07.173169 (+ 1590us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0426 09:55:08.440110 (+1266941us) tablet_metadata.cc:641] Metadata flushed
0426 09:55:08.623520 (+183410us) tablet_metadata.cc:641] Metadata flushed
0426 09:55:08.663539 (+ 40019us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":340412,"tablet-delete.queue_time_us":255758,"thread_start_us":220302,"threads_started":1}
I20240426 09:55:08.735527 11001 raft_consensus.cc:3037] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.739816 11032 raft_consensus.cc:2781] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.730484 10990 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.747750 11014 consensus_queue.cc:237] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:08.749181 10054 consensus_peers.cc:485] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6151686b1a24447880e49b7d7395fcc1. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:08.746613 11017 consensus_queue.cc:1046] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:08.753770 9982 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:08.759330 9825 catalog_manager.cc:4830] TS 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): tablet 21e4aa688b194564bef9b4398f572caf (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:55:08.745810 10992 raft_consensus.cc:729] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:08.763253 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.767481 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.768050 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.771977 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.752786 11032 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:08.781488 11032 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.780537 10936 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.767692 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.784377 11031 raft_consensus.cc:2781] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.785017 11031 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:08.785358 11031 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.767843 11003 raft_consensus.cc:3032] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:08.774333 9906 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:08.753558 11011 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.777144 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.778139 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.779008 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.779095 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.779227 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.779820 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.780025 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.755359 10999 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.780547 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.787834 9979 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:08.773288 10011 raft_consensus.cc:1261] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:55:08.820778 10089 raft_consensus.cc:2379] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:08.816309 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.785739 10990 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.823983 9982 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:08.825229 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.784350 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.824648 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:08.827461 10089 raft_consensus.cc:2379] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:08.828660 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.784653 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:08.815920 11010 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:55:08.833166 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 46670.8us
I20240426 09:55:08.835197 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:08.836580 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.838934 10987 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.836282 11020 raft_consensus.cc:2726] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:08.840466 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.840857 10987 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:08.841784 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 69169.8us
I20240426 09:55:08.842185 10996 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:08.842665 10996 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.843190 10620 raft_consensus.cc:1226] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:08.843847 11032 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.815471 11003 raft_consensus.cc:729] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:08.845436 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.836151 10083 raft_consensus.cc:2379] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
W20240426 09:55:08.846016 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 71947us
I20240426 09:55:08.847193 9979 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:08.836390 11036 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.833220 10999 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.852567 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.846333 10918 consensus_queue.cc:1046] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20240426 09:55:08.825006 11011 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.853804 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:08.845676 10990 raft_consensus.cc:2726] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:08.844952 11010 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.853289 10012 raft_consensus.cc:1226] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:08.857978 9982 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:08.864001 11032 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.845531 10992 consensus_queue.cc:260] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.865346 10992 raft_consensus.cc:3037] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.865200 11001 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.867278 11001 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.868346 10081 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:08.869230 11011 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.851215 11036 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:08.869796 11036 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.856868 10987 raft_consensus.cc:2726] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:08.878976 11032 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.852898 11054 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.884037 10009 raft_consensus.cc:1261] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:08.887327 10007 raft_consensus.cc:1261] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:08.887435 10999 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:08.853642 11038 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.896711 11003 consensus_queue.cc:260] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.897882 11003 raft_consensus.cc:3037] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.897667 10972 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.885425 11032 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:08.899556 11032 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.899608 10972 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.900254 11010 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.903131 11017 raft_consensus.cc:3032] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:08.903527 11017 raft_consensus.cc:729] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:08.904341 9906 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:08.906067 11017 consensus_queue.cc:260] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.911110 11014 consensus_queue.cc:1035] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:08.885530 10936 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.911533 11037 consensus_queue.cc:1035] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20240426 09:55:08.923488 11038 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.926220 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:08.926996 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 100623us
I20240426 09:55:08.927698 9941 raft_consensus.cc:1261] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:08.915779 9906 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:08.917163 9979 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:08.930815 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.934223 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.934114 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.936144 11017 raft_consensus.cc:3037] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.936106 9936 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:08.939913 10620 raft_consensus.cc:1226] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:08.940886 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:08.941612 10613 raft_consensus.cc:2379] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 2: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:08.934181 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.942118 11031 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.943413 10987 raft_consensus.cc:2781] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:08.943715 10987 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:08.943976 10987 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:08.914837 10992 raft_consensus.cc:2726] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:08.891834 10988 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.947182 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.948727 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.950081 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.951848 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.952886 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.931735 11037 consensus_queue.cc:1035] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20240426 09:55:08.960505 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.961328 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.961721 9941 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:08.962216 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:08.953982 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.952919 11011 consensus_queue.cc:1035] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240426 09:55:08.932317 11010 raft_consensus.cc:2726] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:08.972101 9936 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:08.955991 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.972317 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:08.972707 9939 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:08.948091 9906 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:08.963939 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.963912 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.965978 10988 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:08.966784 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.955101 9938 raft_consensus.cc:1226] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:08.973585 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.973809 9935 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
W20240426 09:55:08.975003 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 57932.1us
I20240426 09:55:08.909704 11054 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.974604 11036 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.983722 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.972123 10936 consensus_queue.cc:1046] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20240426 09:55:08.984740 10988 consensus_queue.cc:1046] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:08.985307 11036 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.985510 10583 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:08.985760 10996 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:08.986398 9932 raft_consensus.cc:2379] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:08.987502 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.984755 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.988701 9934 raft_consensus.cc:2379] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:08.988560 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.989424 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:08.989555 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:08.983017 10992 raft_consensus.cc:2726] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:08.991123 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:08.986296 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:08.991456 11037 raft_consensus.cc:2726] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:08.992051 10936 raft_consensus.cc:3032] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:08.992758 9933 raft_consensus.cc:1261] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:08.992879 10936 raft_consensus.cc:729] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:08.988598 9940 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:08.992056 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:08.996028 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:08.997944 10011 raft_consensus.cc:1226] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
W20240426 09:55:08.998304 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 50302.6us
I20240426 09:55:08.999524 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:09.000679 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:09.002743 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:09.003095 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:09.003307 11054 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:09.004283 9906 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:09.006664 10992 raft_consensus.cc:2726] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:08.994431 11037 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20240426 09:55:09.007195 11032 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:08.991796 9934 raft_consensus.cc:1261] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:55:09.009025 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:09.009048 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:09.010340 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.002655 10015 raft_consensus.cc:1226] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:09.014187 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:09.015287 9936 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:09.016249 11032 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:09.019160 10988 consensus_queue.cc:1046] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:09.020123 10090 raft_consensus.cc:1261] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:09.020272 11011 raft_consensus.cc:3032] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:09.020629 11011 raft_consensus.cc:729] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:09.021267 11031 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:09.021406 11011 consensus_queue.cc:260] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:09.021898 10081 raft_consensus.cc:1261] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:09.019371 10085 raft_consensus.cc:1261] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:09.023622 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:09.024387 11011 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:09.024820 10013 raft_consensus.cc:1226] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:08.993733 10936 consensus_queue.cc:260] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:08.987576 11004 raft_consensus.cc:3032] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:09.029624 9906 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:09.033350 11016 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:09.033682 11037 consensus_queue.cc:1035] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 09:55:09.032615 10987 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:09.035387 9906 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:09.031107 11004 raft_consensus.cc:729] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:09.033663 11014 consensus_queue.cc:1035] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20240426 09:55:09.036325 11004 consensus_queue.cc:260] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:09.037302 11010 raft_consensus.cc:2726] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:09.037971 11010 raft_consensus.cc:2726] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:09.039659 10936 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:09.042433 11057 consensus_queue.cc:1046] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 09:55:09.046542 11057 raft_consensus.cc:3032] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:09.047063 11057 raft_consensus.cc:729] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
W20240426 09:55:09.047690 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 66574.9us
I20240426 09:55:09.037374 11004 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:09.071020 10083 raft_consensus.cc:1261] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:09.072867 10015 raft_consensus.cc:1261] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:09.074726 9906 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:09.075917 9906 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:09.078866 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.085819 11057 consensus_queue.cc:260] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:09.072180 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.089370 9979 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:09.090849 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.087100 11057 raft_consensus.cc:3037] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:09.099978 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.078086 11010 raft_consensus.cc:2726] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:09.089490 11017 raft_consensus.cc:2726] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:09.078083 11014 consensus_queue.cc:1035] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20240426 09:55:09.114459 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.115247 11024 consensus_queue.cc:1035] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:09.116818 9935 raft_consensus.cc:2454] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:09.093811 11060 consensus_queue.cc:1046] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20240426 09:55:09.112082 11031 raft_consensus.cc:2726] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:09.113840 10987 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:09.127058 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.171757 10015 raft_consensus.cc:1226] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:09.167804 9979 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:09.172626 9979 raft_consensus.cc:2670] T 21e4aa688b194564bef9b4398f572caf P 826f03b2fb9c4b9483623b041a1bf80e: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240426 09:55:09.124496 11052 raft_consensus.cc:3032] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 2 LEADER]: Stepping down as leader of term 2
W20240426 09:55:09.172168 11064 log.cc:924] Time spent T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.098s user 0.000s sys 0.003s
I20240426 09:55:09.173818 10007 raft_consensus.cc:1261] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240426 09:55:09.174505 11066 log.cc:924] Time spent T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.065s user 0.004s sys 0.000s
I20240426 09:55:09.185025 10999 consensus_queue.cc:1046] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20240426 09:55:09.178737 11052 raft_consensus.cc:729] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:09.195628 11052 consensus_queue.cc:260] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:09.201097 10999 raft_consensus.cc:3032] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:09.210458 9979 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:09.206427 10999 raft_consensus.cc:729] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:09.212407 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.212178 10999 consensus_queue.cc:260] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:09.213302 10999 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:09.191403 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.220470 9979 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:09.221359 9936 raft_consensus.cc:1261] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:09.223435 9979 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:09.223604 10081 raft_consensus.cc:1261] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:09.224675 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.224474 10972 raft_consensus.cc:2781] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:09.225409 10972 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:09.221112 11031 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:09.225768 10972 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:09.225870 11031 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:09.226289 11031 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:09.220458 11052 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:09.233754 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:09.235567 9932 raft_consensus.cc:1261] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 09:55:09.233767 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.224993 10987 raft_consensus.cc:2726] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:09.228751 10016 raft_consensus.cc:2932] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:09.245082 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.238232 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.248821 9979 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:09.251058 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.232582 11014 consensus_queue.cc:1035] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20240426 09:55:09.251365 9979 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:09.252784 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:09.253756 11000 raft_consensus.cc:2781] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:09.254143 11000 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:09.254484 11000 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:09.254319 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:09.255888 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:09.257709 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:09.259038 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:09.261561 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:09.261857 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:09.263600 9938 raft_consensus.cc:2454] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:09.226646 11055 raft_consensus.cc:2932] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:09.266564 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.252130 11036 raft_consensus.cc:2726] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:09.246066 11024 consensus_queue.cc:1035] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20240426 09:55:09.272004 10007 raft_consensus.cc:1226] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:09.272866 11016 consensus_queue.cc:1035] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:09.309491 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:09.323969 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 76314.9us
I20240426 09:55:09.343739 10972 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:09.344782 10007 raft_consensus.cc:1261] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 09:55:09.345664 10972 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:09.347440 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.352880 11037 consensus_queue.cc:1035] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20240426 09:55:09.357343 10013 raft_consensus.cc:1226] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:09.378228 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.381193 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:09.380623 9906 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:09.274964 11075 consensus_queue.cc:1035] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.093s
I20240426 09:55:09.386104 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:09.386992 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:09.387117 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:09.393538 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:09.386643 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:09.407290 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:09.410419 11003 raft_consensus.cc:2726] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 09:55:09.414256 11072 log.cc:924] Time spent T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.139s user 0.001s sys 0.000s
I20240426 09:55:09.410689 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:09.419999 10090 raft_consensus.cc:2454] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:09.421762 10085 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:09.353343 11000 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:09.427481 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:09.432458 10013 raft_consensus.cc:1226] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:09.434867 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:09.435277 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:09.435654 10085 raft_consensus.cc:2379] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:09.437783 10996 consensus_queue.cc:1046] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:09.438683 11028 raft_consensus.cc:3032] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:09.439023 11028 raft_consensus.cc:729] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:09.439359 10013 consensus_queue.cc:237] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:09.439772 11028 consensus_queue.cc:260] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:09.440953 11028 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:09.459180 11079 ts_tablet_manager.cc:902] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:09.462800 9932 raft_consensus.cc:1226] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:09.463725 11000 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:09.467242 10007 raft_consensus.cc:1226] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:09.469259 9979 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:09.472570 9979 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:09.461450 11031 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:09.482331 10016 raft_consensus.cc:1226] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:09.482005 10972 raft_consensus.cc:2781] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:09.487170 11036 raft_consensus.cc:2781] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:09.503317 11036 raft_consensus.cc:686] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:09.504185 11036 consensus_queue.cc:237] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:09.507472 10972 raft_consensus.cc:686] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:09.508369 10972 consensus_queue.cc:237] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:09.512166 11031 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:09.495143 11038 consensus_queue.cc:1046] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20240426 09:55:09.513628 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:09.514192 10936 raft_consensus.cc:3032] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:09.514250 10090 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:09.514748 10936 raft_consensus.cc:729] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:09.502293 11003 consensus_queue.cc:1046] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:09.520673 10087 raft_consensus.cc:3037] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:09.522035 10090 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:09.524017 9979 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:09.524909 10090 raft_consensus.cc:1261] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 09:55:09.515605 10936 consensus_queue.cc:260] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:09.526549 10987 raft_consensus.cc:2781] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:09.527195 10987 raft_consensus.cc:686] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:09.528028 10987 consensus_queue.cc:237] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:09.527204 10936 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:09.531078 9932 raft_consensus.cc:1226] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:09.476437 10996 consensus_queue.cc:1046] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20240426 09:55:09.592705 9839 catalog_manager.cc:5484] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 2 to 3, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 3 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:09.594439 10087 raft_consensus.cc:1261] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20240426 09:55:09.597640 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 114666us
I20240426 09:55:09.596676 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:09.597455 9839 catalog_manager.cc:5484] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 2 to 3, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 3 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 09:55:09.614051 11081 log.cc:924] Time spent T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20240426 09:55:09.601331 10996 raft_consensus.cc:3032] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:09.597311 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:09.644773 10090 raft_consensus.cc:1261] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:55:09.643990 9839 catalog_manager.cc:5484] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 1 to 2, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 2 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:09.625398 11075 consensus_queue.cc:1035] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20240426 09:55:09.659829 9839 catalog_manager.cc:5484] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 1 to 2, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 2 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:09.643173 10996 raft_consensus.cc:729] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:09.663864 11077 consensus_queue.cc:1035] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:09.670478 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:09.678078 11031 consensus_queue.cc:1035] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20240426 09:55:09.726583 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:55:09.734359 9980 consensus_peers.cc:485] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb5d051be8c743a18cd550e145b6564e. This is attempt 1: this message will repeat every 5th retry.
W20240426 09:55:09.734573 11083 log.cc:924] Time spent T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
W20240426 09:55:09.730907 11069 log.cc:924] Time spent T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
W20240426 09:55:09.740614 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 71076.6us
I20240426 09:55:09.725279 10081 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:09.752058 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:09.755415 10084 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:09.740901 10996 consensus_queue.cc:260] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:09.804318 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:09.806370 9938 raft_consensus.cc:2454] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:09.801358 10082 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
W20240426 09:55:09.802026 11086 log.cc:924] Time spent T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.143s user 0.001s sys 0.001s
I20240426 09:55:09.822962 9937 raft_consensus.cc:1226] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:09.827432 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:09.824780 10996 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:09.830758 9936 raft_consensus.cc:1261] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:55:09.831667 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:09.834811 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:09.837184 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:09.872632 10085 raft_consensus.cc:1226] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:09.874300 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:09.875927 10085 raft_consensus.cc:1226] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:09.888456 10088 raft_consensus.cc:1226] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:09.890359 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:09.882894 10086 raft_consensus.cc:1226] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:09.834347 10084 raft_consensus.cc:1226] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:09.895504 10089 raft_consensus.cc:1226] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:09.896045 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:09.864758 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:09.900465 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:09.901212 10992 raft_consensus.cc:3032] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Stepping down as leader of term 1
W20240426 09:55:09.901948 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 161615us
I20240426 09:55:09.905715 9979 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:09.899283 10083 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:09.924249 11036 raft_consensus.cc:2781] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:09.923877 11095 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:09.930696 11095 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:09.932802 11095 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:09.919353 10992 raft_consensus.cc:729] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:09.921993 10972 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:09.933177 10056 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:09.941854 10087 raft_consensus.cc:1226] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:09.933852 11036 raft_consensus.cc:686] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:09.979001 9904 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:09.981248 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 48140us
I20240426 09:55:09.982023 10056 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
W20240426 09:55:09.982682 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 62701.5us
I20240426 09:55:09.973516 11060 raft_consensus.cc:2781] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:09.988898 11060 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:09.989317 11060 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:09.984702 9982 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
W20240426 09:55:09.944015 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 203298us
I20240426 09:55:09.989404 9979 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:09.992671 11054 raft_consensus.cc:2726] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:09.993270 11000 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:09.993945 10992 consensus_queue.cc:260] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:09.995067 10992 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:09.999089 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:09.999722 9937 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:09.943526 10972 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:10.003257 10972 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:10.005023 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:10.005786 10615 raft_consensus.cc:2379] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:10.007521 11020 raft_consensus.cc:2932] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:10.011073 10586 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:10.011798 11016 raft_consensus.cc:2781] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:10.012377 11016 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:10.012678 11016 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:09.983657 11028 raft_consensus.cc:2726] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.015508 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:10.016072 10087 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 1.
I20240426 09:55:10.019766 11000 raft_consensus.cc:686] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:10.020684 11000 consensus_queue.cc:237] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.021009 11016 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:10.023574 11016 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:10.024330 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:10.024892 9937 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:10.027763 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:10.028297 10089 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 3
I20240426 09:55:10.033291 9937 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:55:10.034507 10586 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:10.035356 11080 raft_consensus.cc:2781] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:10.035801 11080 raft_consensus.cc:686] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:10.036600 11080 consensus_queue.cc:237] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:10.039319 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 54696.7us
I20240426 09:55:10.040016 9982 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.041473 10987 raft_consensus.cc:2726] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.042176 9935 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:10.042697 9982 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.043812 9982 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.020821 10083 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:10.046295 10086 raft_consensus.cc:1226] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:10.046927 11061 raft_consensus.cc:2781] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:10.047268 11061 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:10.047899 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 46695.5us
I20240426 09:55:10.048826 9982 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.049932 9982 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.044507 11098 raft_consensus.cc:2726] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.053064 9982 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.059182 11031 raft_consensus.cc:2726] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.047662 11061 raft_consensus.cc:3037] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.045717 10972 raft_consensus.cc:2726] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.063009 9979 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:10.067361 10056 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:10.071738 9904 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.051239 10987 raft_consensus.cc:2726] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.076615 10056 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.076850 9982 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.081316 11028 raft_consensus.cc:2726] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.079087 11038 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:10.083230 11038 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:10.083608 11038 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.086138 10089 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:55:10.079255 11098 raft_consensus.cc:2726] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.081735 10936 raft_consensus.cc:2726] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.088999 9904 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.089824 11028 raft_consensus.cc:2726] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.091434 9904 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.092165 11028 raft_consensus.cc:2726] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.093185 9904 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.078004 11036 consensus_queue.cc:237] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.099519 9982 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.102201 9904 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.113595 10056 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.115839 9982 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.117125 10056 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.117702 9904 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.100693 11028 raft_consensus.cc:2726] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.120076 11055 raft_consensus.cc:2726] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.120874 10056 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:10.121603 10992 raft_consensus.cc:2726] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.121951 10056 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.123912 10056 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.124907 10056 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.120760 10972 consensus_queue.cc:1046] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:10.067420 11060 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.100426 11031 raft_consensus.cc:2726] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.101691 10987 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:10.126888 10056 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.128253 9982 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:10.130121 9904 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.130192 10056 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:10.125877 10936 raft_consensus.cc:2726] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.123230 11032 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.132022 10056 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.133930 9904 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.127331 11055 raft_consensus.cc:2726] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.128643 11011 raft_consensus.cc:2726] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.129349 11028 raft_consensus.cc:2726] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.126996 11054 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:10.139299 9982 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.132308 10999 raft_consensus.cc:2741] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240426 09:55:10.132431 11052 raft_consensus.cc:2726] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.136195 11099 raft_consensus.cc:2726] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.132352 11062 raft_consensus.cc:2726] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.117049 11056 consensus_queue.cc:1046] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20240426 09:55:10.146478 10996 raft_consensus.cc:2726] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.137652 10936 raft_consensus.cc:2726] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.152719 11061 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.088059 10083 raft_consensus.cc:2932] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:10.156277 9904 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:10.154748 11061 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:10.163400 11098 raft_consensus.cc:2781] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:10.163966 11098 raft_consensus.cc:686] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:10.164794 11098 consensus_queue.cc:237] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:10.143801 11054 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:10.171618 11054 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.173310 11062 raft_consensus.cc:2726] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.175143 9904 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.176041 11056 raft_consensus.cc:2741] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240426 09:55:10.178705 9904 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.179838 11028 raft_consensus.cc:3032] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:10.179636 10992 raft_consensus.cc:2726] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.180254 11028 raft_consensus.cc:729] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:10.173507 11060 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:10.189240 9904 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.207935 11038 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.211335 11004 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.184291 10992 raft_consensus.cc:2726] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.213299 11056 raft_consensus.cc:2726] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.217625 9904 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.228066 9904 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.229008 10992 raft_consensus.cc:2726] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.184703 11028 consensus_queue.cc:260] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.234993 11099 raft_consensus.cc:2726] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.235818 11028 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.236114 9904 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.238912 10992 raft_consensus.cc:2726] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.239976 9904 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.240720 10992 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.241766 9904 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.243572 11054 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.245303 11054 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:10.245826 10992 raft_consensus.cc:2726] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.246989 9904 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.247756 10992 raft_consensus.cc:2726] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.248943 9904 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.251253 10992 raft_consensus.cc:2726] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.252295 9904 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.256022 11038 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:10.259230 10992 raft_consensus.cc:2726] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.260264 9904 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.261363 10992 raft_consensus.cc:2726] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.261873 10992 consensus_queue.cc:1046] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:10.248152 11032 raft_consensus.cc:2726] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.262898 11099 raft_consensus.cc:3032] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:10.263242 11099 raft_consensus.cc:729] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:10.279448 10996 consensus_queue.cc:1046] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:10.288673 11056 raft_consensus.cc:3032] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:10.320861 10970 raft_consensus.cc:3032] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:10.321264 10970 raft_consensus.cc:729] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:10.321985 10970 consensus_queue.cc:260] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.323128 10970 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.303185 10996 consensus_queue.cc:1046] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20240426 09:55:10.280123 11099 consensus_queue.cc:260] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.324792 11099 raft_consensus.cc:3037] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.308643 11056 raft_consensus.cc:729] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:10.331396 11056 consensus_queue.cc:260] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.332604 11056 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
W20240426 09:55:10.342226 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 214712us
W20240426 09:55:10.345657 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 206586us
W20240426 09:55:10.346665 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 214712us
I20240426 09:55:10.350355 10056 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.351787 10056 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.353298 10056 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.347715 10987 raft_consensus.cc:686] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:10.351233 9982 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
W20240426 09:55:10.372366 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 85985.6us
I20240426 09:55:10.339212 11028 raft_consensus.cc:3032] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:10.364944 11054 raft_consensus.cc:2726] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.376456 9904 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.376875 10089 raft_consensus.cc:1226] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:10.359095 10936 raft_consensus.cc:2726] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.381914 10086 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:10.384411 9904 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.372085 11011 raft_consensus.cc:2726] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.374434 11028 raft_consensus.cc:729] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:10.446395 11062 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:10.425478 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:10.366000 11004 consensus_queue.cc:1035] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20240426 09:55:10.447016 11062 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:10.449002 11062 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:10.448676 10987 consensus_queue.cc:237] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.453903 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:10.454826 11101 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:10.455279 11101 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:10.456118 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:10.457036 11101 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:10.458151 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:10.380517 11020 raft_consensus.cc:2726] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 09:55:10.474159 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 86920.9us
I20240426 09:55:10.475729 9982 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.483103 10089 raft_consensus.cc:1261] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:10.384617 10992 consensus_queue.cc:1046] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.088s
W20240426 09:55:10.489775 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 105447us
I20240426 09:55:10.472537 10996 raft_consensus.cc:2726] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.485528 11122 raft_consensus.cc:2726] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.495932 10970 raft_consensus.cc:2726] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.483978 11062 consensus_queue.cc:1046] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20240426 09:55:10.498179 10996 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:10.505421 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:10.505218 10992 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:10.535018 9982 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.537616 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:10.538887 11109 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:10.539427 11109 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:10.541524 11110 log.cc:924] Time spent T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.166s user 0.000s sys 0.001s
I20240426 09:55:10.543365 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.544080 9935 raft_consensus.cc:2379] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 2: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:10.508664 11122 raft_consensus.cc:3032] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:10.534152 10996 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.551700 10996 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:10.552203 10996 consensus_queue.cc:1046] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:10.552376 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:10.553666 10082 raft_consensus.cc:1261] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:10.554818 11118 ts_tablet_manager.cc:902] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:10.554819 10086 raft_consensus.cc:1226] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:10.553673 10081 raft_consensus.cc:1226] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
W20240426 09:55:10.560276 9887 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica d3a480e831d649139ff016fb68aee3ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:10.511452 11112 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20240426 09:55:10.569159 11056 raft_consensus.cc:3032] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:10.569546 11056 raft_consensus.cc:729] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:10.571884 11020 raft_consensus.cc:3032] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:10.572263 11020 raft_consensus.cc:729] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:10.572989 11020 consensus_queue.cc:260] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.574146 11020 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.511389 11061 consensus_queue.cc:1046] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20240426 09:55:10.528261 11054 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:10.580580 10011 raft_consensus.cc:1226] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
W20240426 09:55:10.584420 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 49492.1us
I20240426 09:55:10.586217 9982 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.593165 10615 raft_consensus.cc:2379] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:10.594661 11061 raft_consensus.cc:3032] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:10.595052 11061 raft_consensus.cc:729] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:10.595744 11061 consensus_queue.cc:260] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:10.597054 11061 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.599052 11032 raft_consensus.cc:2726] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240426 09:55:10.575275 11031 consensus_queue.cc:1159] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f12521b6d51242629cf4297864df00b9" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20240426 09:55:10.575604 11122 raft_consensus.cc:729] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:10.601001 11122 consensus_queue.cc:260] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.602137 11122 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.548246 10992 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.604522 10992 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:10.503000 11028 consensus_queue.cc:260] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.589198 11054 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:10.606357 10089 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:10.607210 11112 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240426 09:55:10.614277 11109 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:10.584095 11098 raft_consensus.cc:2726] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.615474 10992 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:10.615979 10992 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.617857 10992 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:10.622416 11054 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:10.623453 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:10.624070 10615 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:10.625128 10054 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:10.629818 11028 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.638124 10970 consensus_queue.cc:1046] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:10.639899 10970 raft_consensus.cc:3032] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:10.640252 10970 raft_consensus.cc:729] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:10.641041 10970 consensus_queue.cc:260] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.642143 10970 raft_consensus.cc:3037] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.570293 11056 consensus_queue.cc:260] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.655332 11054 raft_consensus.cc:2781] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:10.655731 11054 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 09:55:10.655687 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 100947us
I20240426 09:55:10.656217 11054 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:10.662838 11054 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:10.665802 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:10.666358 10615 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:10.671314 11079 tablet_copy_client.cc:308] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.131:34821
I20240426 09:55:10.673694 11056 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.676743 10615 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:10.680202 10054 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:10.680936 11127 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:10.681149 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 185532us
I20240426 09:55:10.681491 11127 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.682811 9904 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:10.683128 11124 consensus_queue.cc:1046] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:10.683530 11127 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:10.684020 11127 raft_consensus.cc:2781] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:10.689260 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.689703 11127 raft_consensus.cc:686] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:10.690550 11127 consensus_queue.cc:237] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:10.684712 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 191422us
I20240426 09:55:10.684824 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.693734 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:10.694785 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.695241 9982 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.695688 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:10.696136 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:10.696364 9982 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.696621 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.685853 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.697304 9937 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:10.697640 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:10.697885 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:10.685573 11060 raft_consensus.cc:2781] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:10.698557 11109 consensus_queue.cc:1035] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:10.686530 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.700058 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:10.700816 10081 raft_consensus.cc:2379] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 2: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:10.687227 10996 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:10.701303 10996 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:10.702358 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:10.688254 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.703034 9940 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:10.699025 11060 raft_consensus.cc:686] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:10.687877 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.703560 11112 raft_consensus.cc:2781] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:10.704113 11112 raft_consensus.cc:686] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:10.704928 11112 consensus_queue.cc:237] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.704023 11060 consensus_queue.cc:237] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:10.688365 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.703168 10996 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:10.684171 11124 raft_consensus.cc:3032] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:10.686338 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.706959 9941 raft_consensus.cc:2379] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:10.696837 10089 raft_consensus.cc:2379] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 2: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:10.707288 9936 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:10.687709 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.706934 11124 raft_consensus.cc:729] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:10.708192 9934 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:10.686640 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.700698 11109 raft_consensus.cc:2726] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.709070 11124 consensus_queue.cc:260] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.710033 10081 raft_consensus.cc:2379] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:10.710398 11124 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.712317 10056 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.713454 11115 raft_consensus.cc:2726] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.713845 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:10.714972 9939 raft_consensus.cc:3037] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.718814 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:10.721280 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:10.721570 9939 raft_consensus.cc:1261] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:10.722359 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:10.723109 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.723860 9939 raft_consensus.cc:2454] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:10.725251 9939 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.725486 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:10.725857 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:10.726087 10009 raft_consensus.cc:2454] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:10.726989 11016 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:10.727696 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.727455 11016 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:10.729477 9906 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:10.730468 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:10.730650 9934 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:10.731099 10009 raft_consensus.cc:2454] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 1.
I20240426 09:55:10.731968 10585 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:10.732750 11101 raft_consensus.cc:2781] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:10.733112 11101 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:10.733433 11101 raft_consensus.cc:3037] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:10.737978 11115 consensus_queue.cc:1046] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:10.738504 9934 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.729450 11016 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:10.738957 11011 raft_consensus.cc:3032] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:10.739475 11011 raft_consensus.cc:729] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:10.740319 11011 consensus_queue.cc:260] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.740640 11101 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:10.741657 11011 raft_consensus.cc:3037] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.743425 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:10.744004 10009 raft_consensus.cc:3037] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 3
I20240426 09:55:10.745324 9934 raft_consensus.cc:1261] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:10.747601 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.748243 9934 raft_consensus.cc:2454] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:10.749488 11101 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:10.738360 11054 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:10.752604 11061 consensus_queue.cc:1046] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:10.753723 10972 raft_consensus.cc:3032] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:10.754076 10972 raft_consensus.cc:729] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:10.754352 11098 consensus_queue.cc:1046] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:10.754850 10972 consensus_queue.cc:260] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.755525 11098 raft_consensus.cc:3032] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:10.755795 10972 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.755897 9941 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.756008 11098 raft_consensus.cc:729] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:10.756985 11098 consensus_queue.cc:260] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.758014 11098 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.762478 9941 raft_consensus.cc:1261] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:10.762873 11112 consensus_queue.cc:1035] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:10.770215 9936 raft_consensus.cc:1261] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 09:55:10.771886 10996 raft_consensus.cc:2781] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:10.772330 10996 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:10.772680 10996 raft_consensus.cc:3037] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:10.774814 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.774811 10056 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.776225 11060 raft_consensus.cc:2726] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.777272 10056 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.778558 11060 raft_consensus.cc:2726] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.777576 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:10.779340 10009 raft_consensus.cc:2454] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:55:10.779760 10056 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.782083 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:10.782938 10615 raft_consensus.cc:2379] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:10.783138 10585 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:10.783134 10056 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.784201 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:10.784999 10615 raft_consensus.cc:2354] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
I20240426 09:55:10.785209 10056 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.785624 11054 raft_consensus.cc:2726] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.786103 11011 raft_consensus.cc:2726] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.786368 11057 raft_consensus.cc:2726] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.786710 10056 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.787451 11057 raft_consensus.cc:2726] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.789237 11101 raft_consensus.cc:2781] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:10.789722 11101 raft_consensus.cc:686] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:10.790544 11101 consensus_queue.cc:237] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:10.791072 10615 raft_consensus.cc:3037] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.781803 10996 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:10.793903 9936 raft_consensus.cc:2379] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:10.795114 10996 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:10.796551 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:10.797645 10009 raft_consensus.cc:2379] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:10.798041 9940 raft_consensus.cc:2932] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:10.799268 10586 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.737694 9939 raft_consensus.cc:1261] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:10.808265 11132 consensus_queue.cc:1035] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 09:55:10.809875 10056 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:10.810918 10056 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.811123 11054 raft_consensus.cc:2781] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:10.811311 9982 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:10.811604 11054 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:10.811897 11060 raft_consensus.cc:2726] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:10.812189 11054 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.812381 11112 consensus_queue.cc:1035] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:10.813675 11112 raft_consensus.cc:2781] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:10.814034 11112 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:10.814368 11112 raft_consensus.cc:3037] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.820478 9982 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.821486 11061 raft_consensus.cc:2726] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.823134 11118 tablet_copy_client.cc:308] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.130:44411
I20240426 09:55:10.823515 11098 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:10.822292 11112 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:10.824693 9938 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:10.824857 10026 tablet_copy_service.cc:140] P 826f03b2fb9c4b9483623b041a1bf80e: Received BeginTabletCopySession request for tablet eb5d051be8c743a18cd550e145b6564e from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:36762)
I20240426 09:55:10.825476 10026 tablet_copy_service.cc:161] P 826f03b2fb9c4b9483623b041a1bf80e: Beginning new tablet copy session on tablet eb5d051be8c743a18cd550e145b6564e from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:36762: session id = 1fee3da01ebc463396397057fdb81a68-eb5d051be8c743a18cd550e145b6564e
I20240426 09:55:10.823083 10056 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.827522 9982 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:10.828214 11098 raft_consensus.cc:2741] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240426 09:55:10.829865 9980 leader_election.cc:398] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313) with higher term. Message: Invalid argument: T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
I20240426 09:55:10.831595 9979 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:10.832407 9904 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.831935 10056 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.834218 9904 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.833876 11020 raft_consensus.cc:2781] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:10.835247 10996 raft_consensus.cc:2726] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.833065 11060 raft_consensus.cc:3037] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.834057 11127 raft_consensus.cc:2726] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.835209 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:10.835444 11020 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:10.837165 11020 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.831687 10615 raft_consensus.cc:1261] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:10.838900 9905 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:10.839934 11127 raft_consensus.cc:2726] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:10.843065 9904 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.843750 11127 raft_consensus.cc:2726] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.844133 9904 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.846150 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:10.846833 11127 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:10.832427 9935 raft_consensus.cc:2932] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:10.851116 9982 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:10.854084 10987 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:10.851915 11101 raft_consensus.cc:2726] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.855100 10056 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.854506 10987 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:10.856415 10987 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:10.856987 11116 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:10.857417 11116 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.852067 10936 raft_consensus.cc:2726] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.858709 10026 tablet_copy_source_session.cc:205] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:10.859692 11116 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:10.860321 11011 raft_consensus.cc:2726] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.825676 11061 consensus_queue.cc:1035] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
W20240426 09:55:10.864216 9887 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d9f79bd8994f4b229ca8da8638bb6d5a, TS 826f03b2fb9c4b9483623b041a1bf80e -> TS 1fee3da01ebc463396397057fdb81a68 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:10.864544 9887 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d9f79bd8994f4b229ca8da8638bb6d5a, TS 826f03b2fb9c4b9483623b041a1bf80e -> TS 1fee3da01ebc463396397057fdb81a68 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.866047 10100 tablet_copy_service.cc:140] P d3a480e831d649139ff016fb68aee3ee: Received BeginTabletCopySession request for tablet 6151686b1a24447880e49b7d7395fcc1 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:48202)
I20240426 09:55:10.866672 10100 tablet_copy_service.cc:161] P d3a480e831d649139ff016fb68aee3ee: Beginning new tablet copy session on tablet 6151686b1a24447880e49b7d7395fcc1 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:48202: session id = 1fee3da01ebc463396397057fdb81a68-6151686b1a24447880e49b7d7395fcc1
I20240426 09:55:10.867017 10082 raft_consensus.cc:1261] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 09:55:10.847905 11054 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.870797 11054 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:10.894142 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:10.872084 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:10.895121 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:10.895713 9936 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:10.847884 10972 consensus_queue.cc:1035] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20240426 09:55:10.853428 11020 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:10.899906 10088 raft_consensus.cc:1261] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:10.900516 10056 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:10.901140 10087 raft_consensus.cc:1261] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:10.902894 11054 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:10.891284 11101 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:10.903946 9839 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58806 (request call id 82) took 1392 ms. Trace:
I20240426 09:55:10.855903 10970 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20240426 09:55:10.903213 9836 catalog_manager.cc:5484] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 1 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:10.908735 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:10.909108 10100 tablet_copy_source_session.cc:205] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:10.919286 11020 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:10.919906 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:10.920559 10014 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:10.903264 11054 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:10.920574 11036 consensus_queue.cc:1035] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
W20240426 09:55:10.929598 11141 log.cc:924] Time spent T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20240426 09:55:10.932337 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:10.933107 10615 raft_consensus.cc:2354] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
I20240426 09:55:10.933965 11112 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:10.935060 10055 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:10.936480 11131 consensus_queue.cc:1035] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:10.893373 11061 consensus_queue.cc:1035] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20240426 09:55:10.941800 9980 leader_election.cc:398] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Vote denied by peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313) with higher term. Message: Invalid argument: T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
I20240426 09:55:10.942132 9980 leader_election.cc:401] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 09:55:10.921906 10987 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:10.921703 10012 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:10.947116 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:10.948873 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:10.927073 11054 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:10.964932 10012 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:10.966138 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:10.966882 10090 raft_consensus.cc:2379] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:10.946746 10987 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:10.972965 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:10.924576 11032 consensus_queue.cc:1035] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20240426 09:55:10.906404 9839 rpcz_store.cc:276] 0426 09:55:09.507140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:09.507277 (+ 137us) service_pool.cc:225] Handling call
0426 09:55:10.899622 (+1392345us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:10.073118 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:10.073633 (+ 515us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:10.073669 (+ 36us) write_op.cc:427] Acquired schema lock
0426 09:55:10.073683 (+ 14us) tablet.cc:660] Decoding operations
0426 09:55:10.074255 (+ 572us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:10.074312 (+ 57us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:10.074345 (+ 33us) tablet.cc:683] Acquiring locks for 4 operations
0426 09:55:10.075301 (+ 956us) tablet.cc:699] Row locks acquired
0426 09:55:10.075318 (+ 17us) write_op.cc:260] PREPARE: finished
0426 09:55:10.075453 (+ 135us) write_op.cc:270] Start()
0426 09:55:10.075568 (+ 115us) write_op.cc:276] Timestamp: P: 1714125310075408 usec, L: 0
0426 09:55:10.075586 (+ 18us) op_driver.cc:331] REPLICATION: starting
0426 09:55:10.139521 (+ 63935us) log.cc:842] Serialized 1685 byte log entry
0426 09:55:10.441665 (+302144us) op_driver.cc:444] REPLICATION: finished
0426 09:55:10.787445 (+345780us) write_op.cc:301] APPLY: starting
0426 09:55:10.787519 (+ 74us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:10.787654 (+ 135us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:10.787752 (+ 98us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:10.794188 (+ 6436us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:10.794345 (+ 157us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0426 09:55:10.794391 (+ 46us) write_op.cc:312] APPLY: finished
0426 09:55:10.849458 (+ 55067us) log.cc:842] Serialized 40 byte log entry
0426 09:55:10.855415 (+ 5957us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:10.873037 (+ 17622us) write_op.cc:446] Released schema lock
0426 09:55:10.875281 (+ 2244us) write_op.cc:341] FINISH: Updating metrics
0426 09:55:10.897329 (+ 22048us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:10.898046 (+ 717us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":328792,"apply.run_cpu_time_us":2951,"apply.run_wall_time_us":118112,"num_ops":4,"prepare.queue_time_us":325693,"prepare.run_cpu_time_us":3854,"prepare.run_wall_time_us":72967,"replication_time_us":311088,"thread_start_us":546640,"threads_started":2}]]}
I20240426 09:55:10.951051 11116 raft_consensus.cc:2726] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.914217 11101 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:10.965827 10972 raft_consensus.cc:2726] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313) with higher term. Message: Invalid argument: T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
I20240426 09:55:10.974589 11060 raft_consensus.cc:2726] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:10.984843 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:11.009999 10085 raft_consensus.cc:1226] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:11.020802 9940 raft_consensus.cc:1261] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:11.029992 11118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5d051be8c743a18cd550e145b6564e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:11.030958 11109 consensus_queue.cc:1035] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:11.033941 9934 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:11.035194 9982 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:11.051270 11079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6151686b1a24447880e49b7d7395fcc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:11.048816 11032 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:11.053784 10615 raft_consensus.cc:2379] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
W20240426 09:55:11.068754 11145 log.cc:924] Time spent T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
W20240426 09:55:11.064271 11148 log.cc:924] Time spent T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.151s user 0.001s sys 0.000s
I20240426 09:55:11.067159 11032 raft_consensus.cc:686] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:11.079883 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:11.085449 11150 log.cc:924] Time spent T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
W20240426 09:55:11.092294 9887 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c75e7dd8bbe64a3eb2174cd4a218d51c, TS f12521b6d51242629cf4297864df00b9 -> TS 1fee3da01ebc463396397057fdb81a68 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:11.092597 9887 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet c75e7dd8bbe64a3eb2174cd4a218d51c, TS f12521b6d51242629cf4297864df00b9 -> TS 1fee3da01ebc463396397057fdb81a68 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:11.092725 11032 consensus_queue.cc:237] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:11.097661 9936 raft_consensus.cc:2454] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:11.098563 11101 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:11.102970 9982 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:11.103935 10970 raft_consensus.cc:2781] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:11.104357 10970 raft_consensus.cc:686] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:11.092204 10008 raft_consensus.cc:1261] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:11.104570 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:11.104636 11127 raft_consensus.cc:2726] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:11.114234 10970 consensus_queue.cc:237] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:11.116093 10996 consensus_queue.cc:1035] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:11.116359 9940 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:11.122159 10586 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:11.122833 11014 raft_consensus.cc:2726] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:11.124347 11061 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:11.124387 11054 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:11.130219 11118 tablet_copy_client.cc:783] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:11.131317 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:11.131896 9940 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 3
I20240426 09:55:11.138243 9940 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:11.149139 10056 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:11.130601 11054 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:11.163632 11061 consensus_queue.cc:1046] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240426 09:55:11.173813 11098 raft_consensus.cc:3032] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:11.174590 11098 raft_consensus.cc:729] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:11.176529 11098 consensus_queue.cc:260] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:11.177821 11098 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:11.193301 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:11.193853 10012 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
W20240426 09:55:11.191797 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 222356us
I20240426 09:55:11.137584 11118 tablet_copy_client.cc:647] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:11.202275 10090 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:11.206332 10012 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:11.188163 11115 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:11.218215 11115 raft_consensus.cc:686] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
W20240426 09:55:11.323290 11147 log.cc:924] Time spent T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20240426 09:55:11.373335 11115 consensus_queue.cc:237] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:11.335666 11037 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:11.355870 10996 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:11.374835 10084 raft_consensus.cc:1261] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:11.454901 11037 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:11.442924 11144 log.cc:924] Time spent T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20240426 09:55:11.442108 10996 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:11.496760 11037 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:11.498000 11152 log.cc:924] Time spent T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.215s user 0.000s sys 0.000s
W20240426 09:55:11.499404 11164 log.cc:924] Time spent T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
W20240426 09:55:11.500694 11166 log.cc:924] Time spent T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.128s user 0.000s sys 0.000s
I20240426 09:55:11.472956 10987 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20240426 09:55:11.487192 10936 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:11.483047 11127 consensus_queue.cc:1035] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
W20240426 09:55:11.506112 11155 log.cc:924] Time spent T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.283s user 0.000s sys 0.001s
W20240426 09:55:11.521485 11157 log.cc:924] Time spent T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.069s user 0.000s sys 0.001s
W20240426 09:55:11.522969 11156 log.cc:924] Time spent T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.191s user 0.001s sys 0.000s
I20240426 09:55:11.527823 10996 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:55:11.529582 11159 log.cc:924] Time spent T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.198s user 0.000s sys 0.001s
I20240426 09:55:11.530671 10615 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 09:55:11.508241 10936 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:11.569317 10936 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:11.575927 9941 raft_consensus.cc:1261] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 09:55:11.600296 11169 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 09:55:11.605156 10009 raft_consensus.cc:1261] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 09:55:11.608990 11099 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:11.609573 11099 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:11.613529 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:11.614210 10090 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:11.615280 9904 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:11.616686 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:11.619606 10620 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:11.635435 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:11.636240 9941 raft_consensus.cc:2379] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:11.637701 9905 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:11.611560 11099 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:11.664836 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:11.665658 10009 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:11.618700 11124 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:11.666266 11124 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:11.666613 11124 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:11.667706 11079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:45570 (request call id 100) took 2344 ms. Trace:
I20240426 09:55:11.668108 11079 rpcz_store.cc:276] 0426 09:55:09.323242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:09.323380 (+ 138us) service_pool.cc:225] Handling call
0426 09:55:09.459479 (+136099us) ts_tablet_manager.cc:903] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
0426 09:55:11.024144 (+1564665us) tablet_copy_client.cc:353] Tablet copy session begun
0426 09:55:11.122268 (+ 98124us) tablet_metadata.cc:641] Metadata flushed
0426 09:55:11.122480 (+ 212us) tablet_copy_client.cc:482] Wrote new tablet metadata
0426 09:55:11.630874 (+508394us) tablet_copy_client.cc:403] Wrote new consensus metadata
0426 09:55:11.667209 (+ 36335us) tablet_replica.cc:205] Creating consensus instance
0426 09:55:11.667567 (+ 358us) ts_tablet_manager.cc:958] Replica registered.
0426 09:55:11.667625 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":308,"tablet-copy.queue_time_us":113456,"thread_start_us":1217473,"threads_started":5}
I20240426 09:55:11.669404 11079 tablet_copy_client.cc:783] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:11.669914 11079 tablet_copy_client.cc:647] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:11.671636 11016 consensus_queue.cc:1035] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20240426 09:55:11.686686 11118 tablet_copy_client.cc:523] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:11.671852 11127 raft_consensus.cc:2781] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:11.687809 11127 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:11.688171 11127 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:11.694947 11037 consensus_queue.cc:1035] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240426 09:55:11.702385 11118 tablet_bootstrap.cc:492] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:11.733804 11124 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:11.795235 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 283003us
W20240426 09:55:11.817183 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 328220us
I20240426 09:55:11.858433 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:11.860078 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:11.839989 11099 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:11.859735 10996 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:11.863982 11031 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:11.872540 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:11.886924 11124 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:11.871865 11099 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:11.978529 11124 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:11.980415 11124 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:11.980752 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 162511us
W20240426 09:55:11.981107 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 280294us
I20240426 09:55:11.982401 11124 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:11.888087 11031 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:11.983750 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:11.984290 11031 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:11.984987 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:11.985596 10090 raft_consensus.cc:2454] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:11.878917 10996 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:11.986907 9904 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:11.988165 10996 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:11.988713 10996 raft_consensus.cc:2781] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:11.989079 10996 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:11.989405 10996 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:11.944881 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:11.982059 11127 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:11.945569 10009 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:12.024932 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:12.028229 10083 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:12.027869 10090 raft_consensus.cc:1261] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:12.029331 10081 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:12.037330 11099 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:12.038869 11124 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.039309 11124 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.039812 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:12.043586 11127 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:12.044144 11124 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:12.038983 9904 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:12.073297 9933 raft_consensus.cc:1261] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:12.076675 10083 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:12.066499 11099 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:12.065625 11127 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:55:12.084223 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 219979us
W20240426 09:55:12.093674 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 387549us
I20240426 09:55:12.106624 9904 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:12.084256 11099 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:12.112721 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:12.100986 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:12.094863 11127 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.132565 11127 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:12.133038 11127 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.133440 11127 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.135196 9938 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:12.107807 11181 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.141472 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:12.120769 11099 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:12.170418 10015 raft_consensus.cc:1261] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:12.269160 10011 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 09:55:12.119217 11167 consensus_queue.cc:1035] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.083s
I20240426 09:55:12.201850 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:12.345276 11170 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
W20240426 09:55:12.214392 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 75375us
I20240426 09:55:12.199481 10081 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:12.263804 10970 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:12.263300 11060 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.173840 11011 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.360654 11036 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.354385 11127 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:12.260179 11181 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.364001 11061 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:12.328938 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 67263us
I20240426 09:55:12.365567 10613 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Advancing to term 3
I20240426 09:55:12.355687 11098 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:12.386627 11127 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:12.387449 11183 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.368032 10996 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.371317 11170 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.372645 10970 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.373318 11011 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.374409 11060 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.377389 10585 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:12.376675 11036 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.388036 11061 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.454617 9941 raft_consensus.cc:1226] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:12.398886 11180 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.402803 11187 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:12.461212 11167 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.461624 11187 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.461827 11167 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.463455 11170 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:12.463558 11187 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:12.389907 11098 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.451649 11183 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.463935 11167 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:12.464550 11185 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:12.464906 11194 log.cc:924] Time spent T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20240426 09:55:12.465358 11098 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:12.465036 11185 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:12.462103 11191 log.cc:924] Time spent T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20240426 09:55:12.467244 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 763) took 1349 ms. Trace:
I20240426 09:55:12.467520 11183 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:12.468595 11185 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:12.469202 10008 rpcz_store.cc:276] 0426 09:55:11.117920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:11.118852 (+ 932us) service_pool.cc:225] Handling call
0426 09:55:11.119693 (+ 841us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:11.119992 (+ 299us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:11.120019 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:11.579537 (+459518us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b2c00001368. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca1628ea1 00007f9ca16384d2 00007f9ca2c81f01 00007f9ca2c4671a 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742
0426 09:55:11.579945 (+ 408us) log.cc:842] Serialized 26 byte log entry
0426 09:55:12.026593 (+446648us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:12.030568 (+ 3975us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:12.048022 (+ 17454us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:12.466891 (+418869us) raft_consensus.cc:1671] finished
0426 09:55:12.466918 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:12.467162 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":367,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"spinlock_wait_cycles":33966976,"thread_start_us":398387,"threads_started":1,"wal-append.queue_time_us":592529}
W20240426 09:55:12.471186 11193 log.cc:924] Time spent T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
I20240426 09:55:12.426322 11127 raft_consensus.cc:686] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
W20240426 09:55:12.458532 11184 log.cc:924] Time spent T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.473s user 0.000s sys 0.008s
W20240426 09:55:12.482671 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 376211us
W20240426 09:55:12.466078 11195 log.cc:924] Time spent T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.194s user 0.000s sys 0.000s
I20240426 09:55:12.454802 11171 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.479614 11057 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.509533 11185 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:12.516335 11132 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:12.541323 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 168860us
W20240426 09:55:12.518671 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 148596us
I20240426 09:55:12.479194 11180 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:12.548096 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 190749us
I20240426 09:55:12.539230 11199 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:12.551036 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 894) took 1336 ms. Trace:
W20240426 09:55:12.606076 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 55765.2us
I20240426 09:55:12.522536 11171 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:12.606360 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 59314.2us
I20240426 09:55:12.542034 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:12.603946 10085 rpcz_store.cc:276] 0426 09:55:11.215003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:11.215149 (+ 146us) service_pool.cc:225] Handling call
0426 09:55:11.215451 (+ 302us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:11.357395 (+141944us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:11.357423 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:11.361924 (+ 4501us) log.cc:842] Serialized 26 byte log entry
0426 09:55:11.607988 (+246064us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b48000b0c58. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca7aee2e1 00007f9ca7ae53d7 00007f9ca7ae526a 00007f9ca7bd943f 00007f9ca7bd9107 00007f9ca7bed60c 00007f9ca7bed5b8 00007f9ca7bed538 00007f9ca7bed4f3 00007f9ca7bec79c 00007f9ca6b6f8d6
0426 09:55:11.805559 (+197571us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:11.805599 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:11.805673 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:12.269456 (+463783us) spinlock_profiling.cc:243] Waited 41.6 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:55:12.550817 (+281361us) raft_consensus.cc:1671] finished
0426 09:55:12.550847 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:12.550974 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10388,"raft.queue_time_us":117222,"raft.run_cpu_time_us":1294,"raft.run_wall_time_us":272548,"spinlock_wait_cycles":103080064,"thread_start_us":557313,"threads_started":2,"wal-append.queue_time_us":614233}
I20240426 09:55:12.537711 11057 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.545462 11132 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:12.650782 11196 log.cc:924] Time spent T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.197s user 0.000s sys 0.000s
W20240426 09:55:12.651638 11190 log.cc:924] Time spent T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
I20240426 09:55:12.645037 11181 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:12.612654 11199 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.727617 11132 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:12.728264 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 70) took 1031 ms. Trace:
I20240426 09:55:12.542199 11185 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:12.728612 9935 rpcz_store.cc:276] 0426 09:55:11.696827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:11.696977 (+ 150us) service_pool.cc:225] Handling call
0426 09:55:11.697304 (+ 327us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:11.697592 (+ 288us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:11.697619 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:11.697997 (+ 378us) log.cc:842] Serialized 26 byte log entry
0426 09:55:11.877577 (+179580us) raft_consensus.cc:1631] Marking committed up to 4
0426 09:55:11.877610 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:11.877669 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:12.728084 (+850415us) raft_consensus.cc:1671] finished
0426 09:55:12.728104 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:12.728209 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1246,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":2495,"thread_start_us":179286,"threads_started":1,"wal-append.queue_time_us":413076}
W20240426 09:55:12.730162 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 180351us
I20240426 09:55:12.721201 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:12.651101 11182 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.607020 11187 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.722432 10936 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.549808 11127 consensus_queue.cc:237] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.749975 10936 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.753809 10089 raft_consensus.cc:2379] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 2: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
W20240426 09:55:12.725612 11197 log.cc:924] Time spent T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.202s user 0.002s sys 0.004s
I20240426 09:55:12.728683 11199 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:12.745026 10996 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:12.768893 11212 consensus_queue.cc:1035] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:12.727888 10972 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:12.730806 11031 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
W20240426 09:55:12.773983 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 161556us
I20240426 09:55:12.775010 10936 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:12.722656 11183 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.777201 10055 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:12.742903 10970 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:12.743523 11011 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:12.784104 10936 consensus_queue.cc:1035] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:12.793433 11199 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:12.797099 10996 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:12.811236 10996 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.813669 11132 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:12.814195 11132 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:12.815222 11062 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.072s user 0.015s sys 0.044s
W20240426 09:55:12.815541 11062 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.072s user 0.015s sys 0.044s
I20240426 09:55:12.786536 11124 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:12.818022 11124 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.821727 11185 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:12.822219 11217 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:12.822710 11217 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:12.833468 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 93509.7us
I20240426 09:55:12.772593 11036 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:12.840415 11217 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:12.840970 11217 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.777688 11182 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.841382 11217 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.777710 10083 raft_consensus.cc:2454] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:12.794418 11123 raft_consensus.cc:2726] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:12.769924 11187 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.845691 11011 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20240426 09:55:12.842017 11171 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:12.810627 11199 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.779917 10972 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:12.783772 11031 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.793876 11183 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:12.846426 11211 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.958177 11223 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:12.943639 11210 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.958706 11206 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.958340 11211 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.959208 11206 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.957856 11183 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:12.962280 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:12.962703 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:12.962901 10011 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:12.963827 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:12.963766 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:12.964385 10016 raft_consensus.cc:2454] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:12.964555 10013 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:12.965361 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:12.965463 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:12.966022 10016 raft_consensus.cc:2379] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 2: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:12.966753 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:12.963305 10015 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:12.972479 10087 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240426 09:55:12.976547 11205 log.cc:924] Time spent T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.176s user 0.000s sys 0.001s
I20240426 09:55:12.976548 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:12.978879 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:12.979727 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:12.980682 10088 raft_consensus.cc:2354] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
I20240426 09:55:12.951326 11160 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:12.983553 11231 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:12.983755 11226 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:12.984169 11232 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:12.984131 11231 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.977149 10996 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:12.983875 11224 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:12.984328 11226 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.988034 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:12.969337 11210 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.990832 10088 raft_consensus.cc:2379] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:12.988657 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:12.991514 10085 raft_consensus.cc:2379] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
W20240426 09:55:12.989431 11099 consensus_meta.cc:287] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.728s user 0.005s sys 0.005s
I20240426 09:55:12.994089 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:12.997488 11237 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:12.998010 11237 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.981266 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:12.994671 10087 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:12.958737 11223 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:12.982149 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:13.007637 10011 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:12.988106 11057 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:13.008736 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:12.990388 11182 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:13.010877 10088 raft_consensus.cc:2454] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:12.993253 10996 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:13.015893 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:13.016453 10088 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:13.016592 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:13.023500 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:13.027758 11237 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.028272 11237 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:13.028666 11237 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.030380 11237 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.011781 10970 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:13.033154 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:13.008946 10009 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:13.035214 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:12.995033 11099 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.034951 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:13.035655 10012 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:13.035813 10015 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:13.036645 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:13.037278 10016 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:13.037643 9979 leader_election.cc:398] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
I20240426 09:55:13.037940 9979 leader_election.cc:401] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240426 09:55:13.039084 11212 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:13.039299 11170 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 3
I20240426 09:55:13.040566 9979 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:12.987687 11160 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:13.041667 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:13.013319 11062 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:13.018162 11132 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:13.012873 11180 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:13.046486 11211 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.017280 11127 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:13.050110 10088 raft_consensus.cc:2454] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:13.050879 9979 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:13.044945 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:13.019111 11057 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:13.072054 10016 raft_consensus.cc:2454] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:13.074947 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:13.070439 11233 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:13.072194 9979 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:13.078198 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:12.993983 11232 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:13.032343 11171 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:13.002453 11224 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:13.055282 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:13.020248 10996 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.052300 11237 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:13.074970 11057 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:13.066902 11036 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:13.088028 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:13.172153 9906 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:13.131206 9904 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:13.411886 10016 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:13.074885 11217 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.070024 11187 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.411888 9940 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
W20240426 09:55:13.172418 10613 consensus_meta.cc:287] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68: Time spent flushing consensus metadata: real 0.773s user 0.009s sys 0.000s
I20240426 09:55:13.132546 11226 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:13.383630 11095 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:13.449851 11209 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.039s sys 0.019s
W20240426 09:55:13.454663 11209 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.552s user 0.039s sys 0.020s
I20240426 09:55:13.455812 10996 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.456486 10996 raft_consensus.cc:2781] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:13.456938 10996 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:13.457309 10996 raft_consensus.cc:3037] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:13.081175 10972 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:13.411408 11036 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:13.074332 11180 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:13.451315 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:13.082319 11031 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.105397 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:13.105513 11199 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.077890 11062 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.454135 11226 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:13.080039 11211 raft_consensus.cc:2781] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:13.117774 11231 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.049749 10970 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.452193 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:13.140173 11210 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:13.377748 11236 raft_consensus.cc:2781] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:13.460165 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:13.446812 11011 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:13.080495 11127 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.447068 11171 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.112238 11233 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.700260 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:13.458853 11099 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.562117 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:13.532395 10055 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:13.770921 11099 raft_consensus.cc:2781] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:13.771396 11227 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
W20240426 09:55:13.771411 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 239384us
I20240426 09:55:13.771533 11099 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:13.460520 11095 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:13.772449 11099 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:13.772202 11227 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.773056 10055 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:13.771140 10007 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:13.774602 10055 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:13.448558 11223 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.779314 10016 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:13.450559 11217 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:13.593775 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 457585us
I20240426 09:55:13.774169 11211 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:13.782892 11211 raft_consensus.cc:3037] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:13.782053 11217 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.474328 11036 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:13.791378 9940 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:13.672670 10972 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:13.652427 11032 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:13.792348 10011 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:13.512118 11123 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:13.767903 11187 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:13.672832 11098 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:13.786222 11236 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:13.702057 11246 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:13.777537 11011 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:13.451756 10613 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
W20240426 09:55:13.801286 10015 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.006s sys 0.001s
I20240426 09:55:13.779098 11210 raft_consensus.cc:2781] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:13.448446 11237 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.805191 10970 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.792920 11223 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:13.805611 11032 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.798662 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:13.562148 11242 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:13.776962 11226 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:13.672271 11244 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:13.807019 11123 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.809619 11187 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.840957 11217 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.803737 10972 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:13.780608 11180 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:13.817286 10015 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.715s user 0.006s sys 0.001s
I20240426 09:55:13.815860 11057 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:13.840137 11237 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.812543 11236 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:13.802906 11171 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:13.798662 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 422691us
I20240426 09:55:13.815747 11210 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 09:55:13.828459 10613 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51158 (request call id 112) took 1721 ms (client timeout 2090 ms). Trace:
I20240426 09:55:13.808986 11098 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:13.855118 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 723978us
I20240426 09:55:13.854797 11217 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:13.818486 11248 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:13.847730 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:13.848480 11062 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.815550 11037 raft_consensus.cc:2741] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240426 09:55:13.858803 11231 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:13.847592 11011 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:13.856441 11257 raft_consensus.cc:2781] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240426 09:55:13.857200 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 54511.9us
W20240426 09:55:13.876492 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 704409us
I20240426 09:55:13.855041 11239 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:13.864679 11057 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:13.867226 11224 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:13.881805 11095 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:13.885685 11250 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:13.867621 10012 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:13.823078 11246 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.891840 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:13.880720 10055 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:13.891830 9982 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:13.866744 11210 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:13.881341 9906 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:13.830240 11223 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:13.880491 11217 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.886154 11250 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.875581 11232 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:13.829974 11242 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.919467 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:13.919915 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:13.922641 11226 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:13.880012 11233 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.879917 11237 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:13.935310 11246 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:13.890537 11079 tablet_copy_client.cc:523] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:13.887778 11057 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:13.887005 11127 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:13.938506 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 57200.9us
I20240426 09:55:13.881296 11257 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:13.883846 10009 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:13.886528 10082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 72) took 1531 ms. Trace:
I20240426 09:55:13.935771 11169 raft_consensus.cc:2781] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:13.916517 11224 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:13.923692 10972 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:55:13.946637 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 53647.5us
W20240426 09:55:13.870507 10613 rpcz_store.cc:269] 0426 09:55:12.095358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:12.108158 (+ 12800us) service_pool.cc:225] Handling call
0426 09:55:13.817093 (+1708935us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:13.936589 10970 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:13.887246 11248 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.951340 11217 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.936293 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:13.936403 11123 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:13.954970 9906 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:13.955080 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:13.956295 11233 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:13.947090 11226 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:13.947816 10081 raft_consensus.cc:2454] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:13.952016 11057 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:13.952642 11180 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:13.958922 11169 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:13.950366 11251 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:13.896731 11231 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.973038 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:13.956861 11099 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.967875 11217 raft_consensus.cc:2781] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:13.971859 11233 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:13.844913 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:13.980581 9836 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50254 (request call id 188) took 4160 ms. Trace:
I20240426 09:55:13.967520 9906 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:13.955772 11257 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:13.948171 11237 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.916244 11239 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.972321 10055 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:13.963091 10970 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:13.993413 9906 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:13.966799 11032 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.961407 11224 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:13.993180 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:13.973237 11169 raft_consensus.cc:3037] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:13.959900 10996 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:13.975195 11170 consensus_meta.cc:287] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.935s user 0.009s sys 0.000s
I20240426 09:55:13.876822 11244 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:13.974220 11187 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.974114 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:13.976333 11231 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.980698 11099 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.975456 11098 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.980741 11217 raft_consensus.cc:686] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:14.012715 11263 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:14.013140 11242 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.995290 11255 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:13.995560 10970 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
W20240426 09:55:14.004549 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 80988.9us
I20240426 09:55:13.974674 11262 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:13.976514 11251 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:14.018330 11251 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.957551 10082 rpcz_store.cc:276] 0426 09:55:12.355023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:12.355932 (+ 909us) service_pool.cc:225] Handling call
0426 09:55:13.885265 (+1529333us) spinlock_profiling.cc:243] Waited 2.26 ms on lock 0x7b58000b0498. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:55:13.885462 (+ 197us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:13.885641 (+ 179us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:13.885684 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:13.885729 (+ 45us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:13.885761 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:13.885817 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:13.886463 (+ 646us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5203712}
I20240426 09:55:13.973851 11226 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:14.019181 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:14.019727 9982 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:14.000758 11180 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:14.002722 11127 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:14.002347 11248 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:14.030000 10085 raft_consensus.cc:2379] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 2: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:14.013247 11263 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:14.032032 10084 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:14.032235 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:14.032840 10082 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:13.978061 11211 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:14.033972 11252 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:14.015938 11170 raft_consensus.cc:2726] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
I20240426 09:55:14.016093 11224 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:14.014963 11242 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:14.019995 11231 raft_consensus.cc:2781] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:14.039180 9979 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:14.023810 11099 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:14.023201 11232 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:14.028824 11127 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:14.033977 11187 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:14.028646 11180 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:14.028795 11223 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:13.989843 9836 rpcz_store.cc:276] 0426 09:55:09.819828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:09.820713 (+ 885us) service_pool.cc:225] Handling call
0426 09:55:13.980506 (+4159793us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:11.211880 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:11.448042 (+236162us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:11.452907 (+ 4865us) write_op.cc:427] Acquired schema lock
0426 09:55:11.458690 (+ 5783us) tablet.cc:660] Decoding operations
0426 09:55:11.488252 (+ 29562us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:11.493106 (+ 4854us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:11.494166 (+ 1060us) tablet.cc:683] Acquiring locks for 1 operations
0426 09:55:11.508249 (+ 14083us) tablet.cc:699] Row locks acquired
0426 09:55:11.508271 (+ 22us) write_op.cc:260] PREPARE: finished
0426 09:55:11.515346 (+ 7075us) write_op.cc:270] Start()
0426 09:55:11.519631 (+ 4285us) write_op.cc:276] Timestamp: P: 1714125311514155 usec, L: 0
0426 09:55:11.527168 (+ 7537us) op_driver.cc:331] REPLICATION: starting
0426 09:55:11.527765 (+ 597us) log.cc:842] Serialized 598 byte log entry
0426 09:55:12.168463 (+640698us) op_driver.cc:444] REPLICATION: finished
0426 09:55:12.825194 (+656731us) write_op.cc:301] APPLY: starting
0426 09:55:12.832364 (+ 7170us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:12.847004 (+ 14640us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:12.901758 (+ 54754us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:12.902012 (+ 254us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:12.902170 (+ 158us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0426 09:55:12.902216 (+ 46us) write_op.cc:312] APPLY: finished
0426 09:55:12.902407 (+ 191us) log.cc:842] Serialized 22 byte log entry
0426 09:55:13.455106 (+552699us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:13.455194 (+ 88us) write_op.cc:446] Released schema lock
0426 09:55:13.455239 (+ 45us) write_op.cc:341] FINISH: Updating metrics
0426 09:55:13.455502 (+ 263us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:13.455530 (+ 28us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":503885,"apply.run_cpu_time_us":60435,"apply.run_wall_time_us":671488,"num_ops":1,"prepare.queue_time_us":281216,"prepare.run_cpu_time_us":4157,"prepare.run_wall_time_us":321272,"replication_time_us":610961,"thread_start_us":860779,"threads_started":3,"wal-append.queue_time_us":553767}]]}
I20240426 09:55:14.032248 11263 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:14.029911 11262 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:14.080953 11262 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:14.037420 11252 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:14.042171 11231 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:14.028522 11258 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:14.047995 11233 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:14.054033 11187 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:14.047667 11170 raft_consensus.cc:2781] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:14.054250 11127 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:14.118844 11036 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:14.030530 11255 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:14.054477 11232 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:14.056793 11180 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:14.081792 11248 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:14.100688 11250 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:14.044088 11210 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:14.116708 10996 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:14.126660 11170 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:14.047096 11217 consensus_queue.cc:237] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:14.116139 11237 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:14.044798 11242 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:14.123843 11079 tablet_bootstrap.cc:492] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:14.116139 11226 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:14.115196 11211 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:14.134878 10082 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:14.098306 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:14.135839 9905 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.141.132:34313 timed out after 2.090s (SENT) [suppressed 20 similar messages]
I20240426 09:55:14.108640 11224 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:14.121843 11231 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:16.825292 11287 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:16.825773 11287 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:14.121867 11187 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:16.827770 11287 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:14.121724 11262 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:14.149940 11255 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:14.120918 11239 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:14.109158 11223 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:16.831027 11262 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:16.831332 11223 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:14.057559 11099 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:16.833101 11262 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:16.822624 11224 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:14.148698 11170 raft_consensus.cc:3037] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:16.835604 11224 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:14.168607 11036 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:14.170333 11232 leader_election.cc:288] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:16.833123 11223 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:14.142199 11248 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:16.840406 11248 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:14.170194 10970 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:16.842142 10970 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:14.170583 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 892) took 1119 ms. Trace:
W20240426 09:55:14.172648 11241 log.cc:924] Time spent T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.367s user 0.000s sys 0.001s
I20240426 09:55:14.182010 11180 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:14.185523 11242 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:14.187562 11210 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:14.189566 11204 log.cc:924] Time spent T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 1.574s user 0.014s sys 0.005s
I20240426 09:55:16.843638 11231 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:16.844375 11180 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:14.192827 11257 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:16.844923 11187 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:16.846190 11257 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:14.175940 11244 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:14.186196 11236 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:16.847940 11187 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:16.849774 11231 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:16.850178 11236 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:14.178175 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:14.179261 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:16.981050 10082 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 934) took 2885 ms (client timeout 1690 ms). Trace:
I20240426 09:55:14.174353 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:16.982319 10088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 920) took 2943 ms (client timeout 1852 ms). Trace:
I20240426 09:55:14.163232 10081 raft_consensus.cc:2354] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
W20240426 09:55:16.988406 10081 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 766) took 3067 ms (client timeout 1714 ms). Trace:
W20240426 09:55:16.988750 10081 rpcz_store.cc:269] 0426 09:55:13.920917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.972689 (+ 51772us) service_pool.cc:225] Handling call
0426 09:55:16.988218 (+3015529us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b54001815d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:16.988344 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":325771136}
W20240426 09:55:16.989749 10081 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 931) took 2922 ms (client timeout 1842 ms). Trace:
W20240426 09:55:16.990098 10081 rpcz_store.cc:269] 0426 09:55:14.067068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:16.989591 (+2922523us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:16.989695 (+ 104us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:15.122823 11274 log.cc:924] Time spent T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
I20240426 09:55:16.043677 11277 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:16.190888 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 165690us
W20240426 09:55:16.991955 9906 connection.cc:232] RPC call timeout handler was delayed by -1.3266991500000245s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29560000000000003s for a response
I20240426 09:55:16.991717 11277 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:16.992324 9906 connection.cc:232] RPC call timeout handler was delayed by -1.311778855s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1714s for a response
W20240426 09:55:16.992623 9906 connection.cc:232] RPC call timeout handler was delayed by -1.0903788549999831s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.196s for a response
I20240426 09:55:16.993813 11277 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:16.384794 10620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45570 (request call id 114) took 3838 ms. Trace:
I20240426 09:55:16.617404 11280 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
W20240426 09:55:16.658537 10055 connection.cc:232] RPC call timeout handler was delayed by -2.1624188000000117s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1632s for a response
I20240426 09:55:16.995247 11253 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:16.995108 11280 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:16.744271 11281 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:16.995774 11253 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:16.996232 11281 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:16.997246 11280 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:16.997797 11054 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:16.998075 11253 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:16.801471 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:16.803794 9982 connection.cc:232] RPC call timeout handler was delayed by -2.6281721809999681s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16920000000000002s for a response
I20240426 09:55:16.998327 11054 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:16.998885 10008 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 767) took 4038 ms (client timeout 1797 ms). Trace:
I20240426 09:55:16.803319 11283 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:16.806210 10011 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1110) took 3771 ms (client timeout 1632 ms). Trace:
W20240426 09:55:16.806612 9979 connection.cc:232] RPC call timeout handler was delayed by -2.0957591970000067s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18520000000000003s for a response
I20240426 09:55:16.806826 11285 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:17.000115 11183 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:17.000336 11054 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:55:17.000008 9979 connection.cc:232] RPC call timeout handler was delayed by -1.6111673730000007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1995s for a response
I20240426 09:55:17.000772 11098 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:17.000901 11054 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:17.001379 11098 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:16.813751 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:16.819478 10085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 772) took 2897 ms (client timeout 1912 ms). Trace:
I20240426 09:55:16.851871 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.002058 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.002486 10007 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
W20240426 09:55:17.002964 10007 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 779) took 3608 ms (client timeout 2239 ms). Trace:
I20240426 09:55:16.999595 11283 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:17.003325 10007 rpcz_store.cc:269] 0426 09:55:13.394121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.829173 (+435052us) service_pool.cc:225] Handling call
0426 09:55:16.851727 (+3022554us) spinlock_profiling.cc:243] Waited 6.84 ms on lock 0x7b580004ec98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:55:17.002905 (+151178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15723264}
I20240426 09:55:17.003911 11098 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:16.862485 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:16.877090 9940 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1097) took 3820 ms (client timeout 1632 ms). Trace:
I20240426 09:55:16.877825 10090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 901) took 4846 ms. Trace:
W20240426 09:55:16.881757 9935 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.730s user 0.006s sys 0.031s
I20240426 09:55:16.882826 9936 raft_consensus.cc:1261] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:16.883348 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:16.884613 9904 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.937s user 0.015s sys 0.027s
W20240426 09:55:17.004894 10055 connection.cc:232] RPC call timeout handler was delayed by -1.048360783000021s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1784s for a response
W20240426 09:55:16.890326 10614 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51158 (request call id 113) took 4047 ms (client timeout 1829 ms). Trace:
W20240426 09:55:17.004981 9979 connection.cc:232] RPC call timeout handler was delayed by -1.6030673729999876s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20040000000000002s for a response
W20240426 09:55:17.005455 10614 rpcz_store.cc:269] 0426 09:55:12.843158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:16.890128 (+4046970us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:16.890250 (+ 122us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:16.891537 9941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 887) took 4792 ms. Trace:
W20240426 09:55:16.893059 10586 connection.cc:232] RPC call timeout handler was delayed by -2.4897872910000274s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18220000000000003s for a response
I20240426 09:55:16.899655 10013 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1109) took 3865 ms. Trace:
W20240426 09:55:17.006386 9979 connection.cc:232] RPC call timeout handler was delayed by -1.339828639000018s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18420000000000003s for a response
W20240426 09:55:17.004938 9935 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.853s user 0.006s sys 0.032s
W20240426 09:55:17.002893 10085 rpcz_store.cc:269] 0426 09:55:13.922339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:16.819233 (+2896894us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:16.819382 (+ 149us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:16.909073 9980 connection.cc:232] RPC call timeout handler was delayed by -2.4266545649999784s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1676s for a response
I20240426 09:55:16.912434 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 71) took 4093 ms. Trace:
I20240426 09:55:16.916023 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.009341 9939 raft_consensus.cc:2354] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
I20240426 09:55:16.917634 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
W20240426 09:55:16.919952 10089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 758) took 4400 ms (client timeout 2803 ms). Trace:
I20240426 09:55:16.920784 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:16.926959 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:14.210094 11057 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:16.860924 11099 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:16.998281 11281 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:16.904461 11124 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:14.432858 11169 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:14.228390 11127 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:17.016031 9979 connection.cc:232] RPC call timeout handler was delayed by -1.3124136970000109s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.169s for a response
I20240426 09:55:17.016258 11169 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:17.016544 11127 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:16.106369 11132 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:16.198643 11060 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:16.093461 11061 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:17.018555 10081 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 768) took 3093 ms (client timeout 2117 ms). Trace:
I20240426 09:55:15.267997 11079 tablet_bootstrap.cc:700] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: Time spent opening tablet: real 0.158s user 0.003s sys 0.000s
I20240426 09:55:17.022433 11057 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:17.007863 10090 rpcz_store.cc:276] 0426 09:55:12.031126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:12.031293 (+ 167us) service_pool.cc:225] Handling call
0426 09:55:12.031559 (+ 266us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:12.077850 (+ 46291us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:12.077877 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:12.078267 (+ 390us) log.cc:842] Serialized 26 byte log entry
0426 09:55:12.456543 (+378276us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:12.456583 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:12.456651 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:14.189099 (+1732448us) spinlock_profiling.cc:243] Waited 35.4 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:55:16.877510 (+2688411us) raft_consensus.cc:1671] finished
0426 09:55:16.877531 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:16.877748 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53215,"raft.queue_time_us":46598,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":8894,"spinlock_wait_cycles":81517440,"thread_start_us":356177,"threads_started":1,"wal-append.queue_time_us":450364}
W20240426 09:55:17.023598 10090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 925) took 2979 ms (client timeout 2004 ms). Trace:
W20240426 09:55:17.023909 10090 rpcz_store.cc:269] 0426 09:55:14.043644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.023429 (+2979785us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.023543 (+ 114us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.024572 10090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 935) took 2922 ms (client timeout 2028 ms). Trace:
W20240426 09:55:17.024859 10090 rpcz_store.cc:269] 0426 09:55:14.101680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.024413 (+2922733us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.024498 (+ 85us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.025485 10090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 775) took 3102 ms (client timeout 2207 ms). Trace:
W20240426 09:55:17.025766 10090 rpcz_store.cc:269] 0426 09:55:13.923412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.025356 (+3101944us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.025442 (+ 86us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.008404 9940 rpcz_store.cc:269] 0426 09:55:13.057083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.451074 (+393991us) service_pool.cc:225] Handling call
0426 09:55:16.876910 (+3425836us) spinlock_profiling.cc:243] Waited 84.5 ms on lock 0x7b54001fd454. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:16.877008 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":194340224}
W20240426 09:55:17.027974 9940 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 904) took 3974 ms (client timeout 1692 ms). Trace:
W20240426 09:55:17.028290 9940 rpcz_store.cc:269] 0426 09:55:13.053756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.027797 (+3974041us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.027906 (+ 109us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.028987 9940 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 902) took 3975 ms (client timeout 2064 ms). Trace:
W20240426 09:55:17.029258 9940 rpcz_store.cc:269] 0426 09:55:13.053316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.028849 (+3975533us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.028943 (+ 94us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.029739 9838 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50252 (request call id 188) took 7107 ms. Trace:
W20240426 09:55:17.026415 10090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 765) took 3105 ms (client timeout 2239 ms). Trace:
I20240426 09:55:14.213629 11118 tablet_bootstrap.cc:492] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:17.032215 11290 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:17.032840 11118 tablet_bootstrap.cc:492] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:17.033645 11118 ts_tablet_manager.cc:1367] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 5.334s user 0.023s sys 0.034s
W20240426 09:55:17.034410 9982 connection.cc:232] RPC call timeout handler was delayed by -2.6107916509999427s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19210000000000002s for a response
I20240426 09:55:16.128182 11160 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:16.931099 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:16.863408 11170 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:17.037585 11118 raft_consensus.cc:348] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:17.038138 11170 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:16.800626 11171 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:16.801584 11186 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:17.039294 11171 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:17.040328 10014 raft_consensus.cc:2379] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:16.900569 11210 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:17.043617 11210 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:17.045459 11210 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:16.820300 11212 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:17.046527 11212 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:17.048291 11212 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:16.900950 11180 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:17.049348 11180 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:17.051185 11180 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:15.900117 11214 log.cc:924] Time spent T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 3.148s user 0.010s sys 0.028s
I20240426 09:55:16.901404 11232 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:17.054147 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 76) took 4881 ms. Trace:
I20240426 09:55:16.866122 11236 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
W20240426 09:55:15.346437 11263 consensus_meta.cc:287] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 1.225s user 0.009s sys 0.000s
I20240426 09:55:17.056514 11263 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:17.058303 11263 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:16.868803 11246 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:17.059352 11246 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:14.221227 11252 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:14.120633 11258 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:17.063825 11258 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:16.864322 11227 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:16.896337 11257 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:17.065856 11257 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:16.868804 11187 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:16.950501 10083 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 759) took 4431 ms (client timeout 2211 ms). Trace:
W20240426 09:55:17.069705 10083 rpcz_store.cc:269] 0426 09:55:12.519163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:12.524020 (+ 4857us) service_pool.cc:225] Handling call
0426 09:55:16.950298 (+4426278us) spinlock_profiling.cc:243] Waited 50.7 ms on lock 0x7b5400251f54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:16.950420 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":116606720}
W20240426 09:55:17.070806 10083 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 770) took 3148 ms (client timeout 2243 ms). Trace:
W20240426 09:55:17.071151 10083 rpcz_store.cc:269] 0426 09:55:13.921860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.070607 (+3148747us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.070727 (+ 120us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.071918 10083 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 933) took 2986 ms (client timeout 2230 ms). Trace:
W20240426 09:55:17.072256 10083 rpcz_store.cc:269] 0426 09:55:14.085208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.071770 (+2986562us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.071867 (+ 97us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.073024 10083 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 936) took 2965 ms (client timeout 2230 ms). Trace:
W20240426 09:55:17.073344 10083 rpcz_store.cc:269] 0426 09:55:14.107441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.072873 (+2965432us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.072970 (+ 97us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.074131 10083 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 924) took 3030 ms (client timeout 2621 ms). Trace:
I20240426 09:55:17.069725 11187 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:17.076836 11187 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:16.959362 10012 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 781) took 3564 ms (client timeout 1960 ms). Trace:
I20240426 09:55:16.962421 10015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1121) took 3917 ms. Trace:
I20240426 09:55:17.078060 10015 rpcz_store.cc:276] 0426 09:55:13.044845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.859253 (+814408us) service_pool.cc:225] Handling call
0426 09:55:16.960043 (+3100790us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:16.960214 (+ 171us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:16.960238 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:16.960275 (+ 37us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:16.960303 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:16.960354 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:16.962261 (+ 1907us) spinlock_profiling.cc:243] Waited 3.1 s on lock 0x7b540025ec50. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c45b99 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:16.962367 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7130744576}
W20240426 09:55:17.079339 10015 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 793) took 3066 ms (client timeout 1593 ms). Trace:
W20240426 09:55:17.079650 10015 rpcz_store.cc:269] 0426 09:55:14.012428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.079198 (+3066770us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.079287 (+ 89us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.080332 10015 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 790) took 3079 ms (client timeout 1608 ms). Trace:
W20240426 09:55:17.080616 10015 rpcz_store.cc:269] 0426 09:55:14.001358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.080183 (+3078825us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.080268 (+ 85us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.081219 10015 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 796) took 3052 ms (client timeout 1627 ms). Trace:
W20240426 09:55:17.081486 10015 rpcz_store.cc:269] 0426 09:55:14.029085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.081095 (+3052010us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.081179 (+ 84us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.082090 10015 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 800) took 2959 ms (client timeout 1657 ms). Trace:
W20240426 09:55:17.082357 10015 rpcz_store.cc:269] 0426 09:55:14.122362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.081969 (+2959607us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.082051 (+ 82us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:16.967599 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:16.973202 9835 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50240 (request call id 198) took 6277 ms. Trace:
I20240426 09:55:17.084352 9835 rpcz_store.cc:276] 0426 09:55:10.695249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:10.695446 (+ 197us) service_pool.cc:225] Handling call
0426 09:55:10.798526 (+103080us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54000041d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcbc70 00007f9ca2c4f0ba 00007f9ca8ba096c 00007f9ca8c9a8de 00007f9ca6faea04 00007f9ca6fae992 00007f9ca6fae8c1 00007f9ca6fae83c 00007f9ca6fadaf2 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:16.973027 (+6174501us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":4,"spinlock_wait_cycles":231577856}
W20240426 09:55:16.981432 10082 rpcz_store.cc:269] 0426 09:55:14.095544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:14.175820 (+ 80276us) service_pool.cc:225] Handling call
0426 09:55:16.980963 (+2805143us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:55:17.086179 10082 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 932) took 3003 ms (client timeout 2870 ms). Trace:
W20240426 09:55:17.086534 10082 rpcz_store.cc:269] 0426 09:55:14.082478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.086023 (+3003545us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.086126 (+ 103us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:16.982255 10016 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 782) took 3587 ms (client timeout 1714 ms). Trace:
I20240426 09:55:16.981439 10009 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
W20240426 09:55:17.077383 10012 rpcz_store.cc:269] 0426 09:55:13.394564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.699961 (+305397us) service_pool.cc:225] Handling call
0426 09:55:16.959175 (+3259214us) spinlock_profiling.cc:243] Waited 97.3 ms on lock 0x7b540025ec54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:16.959292 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":223700864}
W20240426 09:55:17.087582 10009 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 789) took 3088 ms (client timeout 1520 ms). Trace:
W20240426 09:55:16.982722 10088 rpcz_store.cc:269] 0426 09:55:14.039252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:14.174096 (+134844us) service_pool.cc:225] Handling call
0426 09:55:16.982257 (+2808161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:16.985952 9937 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240426 09:55:17.087953 10012 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 801) took 2965 ms (client timeout 1665 ms). Trace:
W20240426 09:55:17.087363 10016 rpcz_store.cc:269] 0426 09:55:13.394807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.451958 (+ 57151us) service_pool.cc:225] Handling call
0426 09:55:16.982077 (+3530119us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b540025fdd4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:16.982192 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24798592}
I20240426 09:55:17.088433 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 897) took 4036 ms. Trace:
I20240426 09:55:17.089419 9937 rpcz_store.cc:276] 0426 09:55:13.052109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.072851 (+ 20742us) service_pool.cc:225] Handling call
0426 09:55:16.985392 (+3912541us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x7b5800004598. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:55:16.985593 (+ 201us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.088182 (+102589us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.088380 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42392832}
I20240426 09:55:17.090453 10082 raft_consensus.cc:1261] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20240426 09:55:17.090596 9937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1099) took 4008 ms (client timeout 2394 ms). Trace:
I20240426 09:55:17.091096 10082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 771) took 3169 ms. Trace:
I20240426 09:55:17.088989 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:17.091864 10088 raft_consensus.cc:3037] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
W20240426 09:55:16.985905 10084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 763) took 3070 ms (client timeout 2036 ms). Trace:
W20240426 09:55:17.088331 10012 rpcz_store.cc:269] 0426 09:55:14.122679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.087806 (+2965127us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.087900 (+ 94us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.092784 10012 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 795) took 3064 ms (client timeout 1847 ms). Trace:
W20240426 09:55:17.092473 10084 rpcz_store.cc:269] 0426 09:55:13.915776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.919951 (+ 4175us) service_pool.cc:225] Handling call
0426 09:55:16.985727 (+3065776us) spinlock_profiling.cc:243] Waited 65.3 ms on lock 0x7b54000daa54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:16.985836 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":150075136}
I20240426 09:55:16.999984 11285 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:16.810034 10086 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 915) took 3795 ms. Trace:
I20240426 09:55:17.001499 11054 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:16.842703 9934 rpcz_store.cc:276] 0426 09:55:13.050846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.052271 (+ 1425us) service_pool.cc:225] Handling call
0426 09:55:13.075301 (+ 23030us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:13.459863 (+384562us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 09:55:13.494875 (+ 35012us) log.cc:842] Serialized 14 byte log entry
0426 09:55:13.802050 (+307175us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:13.802105 (+ 55us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:13.802154 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:14.170335 (+368181us) spinlock_profiling.cc:243] Waited 378 ms on lock 0x7b540019c154. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:14.170385 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:14.170508 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":7942,"prepare.run_wall_time_us":38,"raft.queue_time_us":12711,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":2385,"spinlock_wait_cycles":899071360,"thread_start_us":205599,"threads_started":1,"wal-append.queue_time_us":313902,"wal-append.run_cpu_time_us":903,"wal-append.run_wall_time_us":1008846}
W20240426 09:55:16.859290 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 2.72337e+06us
I20240426 09:55:17.093436 10086 rpcz_store.cc:276] 0426 09:55:13.014292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.014403 (+ 111us) service_pool.cc:225] Handling call
0426 09:55:13.014679 (+ 276us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:13.015007 (+ 328us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:13.015034 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:13.015393 (+ 359us) log.cc:842] Serialized 26 byte log entry
0426 09:55:16.807632 (+3792239us) spinlock_profiling.cc:243] Waited 1.85 ms on lock 0x7b180027f2e0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fcfdb75 00007f9ca8b37789 00007f9c9fd1b0eb 00007f9c9fd194ef 00007f9c9fd1b652 00007f9ca28c743b 00007f9ca28ccee3 00007f9ca28cd291 00007f9ca2c1a75f
0426 09:55:16.808345 (+ 713us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:16.808386 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:16.808451 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:16.809752 (+ 1301us) raft_consensus.cc:1671] finished
0426 09:55:16.809770 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:16.809957 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":85,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":9153,"spinlock_wait_cycles":4246784,"thread_start_us":522,"threads_started":1,"wal-append.queue_time_us":3789733}
W20240426 09:55:17.087975 10009 rpcz_store.cc:269] 0426 09:55:13.999023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:14.178991 (+179968us) service_pool.cc:225] Handling call
0426 09:55:17.087515 (+2908524us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:55:17.095183 9905 connection.cc:232] RPC call timeout handler was delayed by -2.6049626860000217s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1829s for a response
W20240426 09:55:17.004302 10007 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 792) took 2999 ms (client timeout 1585 ms). Trace:
W20240426 09:55:17.002565 10087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 764) took 3082 ms (client timeout 1960 ms). Trace:
I20240426 09:55:17.006666 10013 rpcz_store.cc:276] 0426 09:55:13.033805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.034051 (+ 246us) service_pool.cc:225] Handling call
0426 09:55:13.034299 (+ 248us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:13.785710 (+751411us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:13.785741 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:13.785782 (+ 41us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:13.785810 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:16.899442 (+3113632us) spinlock_profiling.cc:243] Waited 751 ms on lock 0x7b540025f654. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:16.899481 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:16.899584 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1727757696}
W20240426 09:55:16.901320 10056 connection.cc:232] RPC call timeout handler was delayed by -2.8852239440000176s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1632s for a response
W20240426 09:55:17.096575 10056 connection.cc:232] RPC call timeout handler was delayed by -2.4764723110000659s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16210000000000002s for a response
I20240426 09:55:17.010713 9932 raft_consensus.cc:2354] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
W20240426 09:55:17.096889 9934 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1103) took 3966 ms (client timeout 2968 ms). Trace:
W20240426 09:55:17.097209 9932 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 903) took 4043 ms (client timeout 1591 ms). Trace:
W20240426 09:55:17.097687 10013 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 785) took 3101 ms (client timeout 1903 ms). Trace:
I20240426 09:55:17.095633 10086 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 921) took 3056 ms. Trace:
I20240426 09:55:17.004321 11181 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:17.100303 11181 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:17.008754 9980 connection.cc:232] RPC call timeout handler was delayed by -2.165077589999953s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18520000000000003s for a response
I20240426 09:55:17.008988 9938 rpcz_store.cc:276] 0426 09:55:12.819240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:12.823008 (+ 3768us) service_pool.cc:225] Handling call
0426 09:55:12.823327 (+ 319us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:12.824594 (+ 1267us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:12.824621 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:12.824660 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:12.824687 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:16.912195 (+4087508us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x7b54001978d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:16.912236 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:16.912354 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2545920}
W20240426 09:55:17.103256 9938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1096) took 4046 ms (client timeout 3093 ms). Trace:
I20240426 09:55:17.004448 11215 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:55:16.999289 10008 rpcz_store.cc:269] 0426 09:55:12.960881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:12.965688 (+ 4807us) service_pool.cc:225] Handling call
0426 09:55:16.801383 (+3835695us) spinlock_profiling.cc:243] Waited 1.44 ms on lock 0x7b580004ec98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce448c 00007f9c9fce43f6 00000000003399f9 00007f9ca7b13570 00007f9ca7bd9294 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:55:16.998815 (+197432us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3320192}
I20240426 09:55:17.091569 10082 rpcz_store.cc:276] 0426 09:55:13.922096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.089796 (+3167700us) service_pool.cc:225] Handling call
0426 09:55:17.090093 (+ 297us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.090869 (+ 776us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.091046 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:17.103825 11215 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:17.074463 10083 rpcz_store.cc:269] 0426 09:55:14.043472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.073982 (+3030510us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.074082 (+ 100us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.102689 9980 connection.cc:232] RPC call timeout handler was delayed by -1.1787252479999779s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2006s for a response
I20240426 09:55:17.008327 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 893) took 3957 ms. Trace:
I20240426 09:55:16.994925 10620 rpcz_store.cc:276] 0426 09:55:12.603786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:12.642378 (+ 38592us) service_pool.cc:225] Handling call
0426 09:55:12.721519 (+ 79141us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:12.730470 (+ 8951us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:12.730534 (+ 64us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:12.730607 (+ 73us) raft_consensus.cc:1631] Marking committed up to 4
0426 09:55:12.730669 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:16.384463 (+3653794us) spinlock_profiling.cc:243] Waited 8.65 ms on lock 0x7b5400450e54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:16.384526 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:16.384697 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19891584}
W20240426 09:55:17.011633 10085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 926) took 2967 ms (client timeout 1995 ms). Trace:
W20240426 09:55:17.014263 9981 service_pool.cc:135] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34378 dropped due to backpressure. The service queue is full; it has 50 items.
W20240426 09:55:17.107605 10085 rpcz_store.cc:269] 0426 09:55:14.043789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.011456 (+2967667us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.011576 (+ 120us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.108418 10008 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 788) took 3111 ms (client timeout 1912 ms). Trace:
I20240426 09:55:17.103780 11181 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:17.108798 10008 rpcz_store.cc:269] 0426 09:55:13.997365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.108262 (+3110897us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.108361 (+ 99us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.000662 11183 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:17.016906 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 894) took 3965 ms. Trace:
I20240426 09:55:17.006358 9941 rpcz_store.cc:276] 0426 09:55:12.099229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:12.099478 (+ 249us) service_pool.cc:225] Handling call
0426 09:55:12.100033 (+ 555us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:12.460099 (+360066us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:16.891305 (+4431206us) spinlock_profiling.cc:243] Waited 354 ms on lock 0x7b54001649d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:16.891469 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":815162112}
I20240426 09:55:17.105846 9935 rpcz_store.cc:276] 0426 09:55:13.051125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.054162 (+ 3037us) service_pool.cc:225] Handling call
0426 09:55:13.981301 (+927139us) spinlock_profiling.cc:243] Waited 7.85 ms on lock 0x7b5800004598. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:55:13.981467 (+ 166us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:13.981630 (+ 163us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 09:55:14.151311 (+169681us) log.cc:842] Serialized 14 byte log entry
0426 09:55:17.008037 (+2856726us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:17.008085 (+ 48us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:17.008125 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.008183 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.008279 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":189425,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":41,"raft.queue_time_us":4160,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":82,"spinlock_wait_cycles":18065664,"thread_start_us":3025798,"threads_started":2,"wal-append.queue_time_us":52822,"wal-append.run_cpu_time_us":1331,"wal-append.run_wall_time_us":1004163}
W20240426 09:55:17.005275 9904 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.058s user 0.016s sys 0.027s
W20240426 09:55:17.009929 10089 rpcz_store.cc:269] 0426 09:55:12.518870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:12.555562 (+ 36692us) service_pool.cc:225] Handling call
0426 09:55:16.919292 (+4363730us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b5400126454. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:16.919401 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41508096}
I20240426 09:55:17.017896 11132 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:17.111446 9936 rpcz_store.cc:276] 0426 09:55:13.051374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.068909 (+ 17535us) service_pool.cc:225] Handling call
0426 09:55:16.882378 (+3813469us) spinlock_profiling.cc:243] Waited 4.13 ms on lock 0x7b5800004598. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:55:16.882538 (+ 160us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.016683 (+134145us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.016860 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9496960}
W20240426 09:55:17.019064 10081 rpcz_store.cc:269] 0426 09:55:13.921399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:16.990673 (+3069274us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.014731 (+ 24058us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.020401 9979 connection.cc:232] RPC call timeout handler was delayed by -1.0477673729999992s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2621s for a response
I20240426 09:55:17.004745 11283 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:17.010205 9933 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 898) took 3957 ms (client timeout 1921 ms). Trace:
W20240426 09:55:17.009748 9939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 899) took 3957 ms (client timeout 1663 ms). Trace:
W20240426 09:55:17.030468 10090 rpcz_store.cc:269] 0426 09:55:13.920676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.026286 (+3105610us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.026372 (+ 86us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.126127 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:17.029892 9940 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 901) took 3976 ms (client timeout 2364 ms). Trace:
W20240426 09:55:17.034826 9982 connection.cc:232] RPC call timeout handler was delayed by -2.4778971289999845s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20640000000000003s for a response
I20240426 09:55:17.030086 9838 rpcz_store.cc:276] 0426 09:55:09.922706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:09.927751 (+ 5045us) service_pool.cc:225] Handling call
0426 09:55:17.029589 (+7101838us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42516,"reported_tablets":5}
I20240426 09:55:17.038465 11118 raft_consensus.cc:729] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:17.130395 11118 consensus_queue.cc:260] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:17.039873 11186 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:16.795724 11206 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:17.032801 11290 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:17.042569 10014 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 769) took 4081 ms (client timeout 2956 ms). Trace:
W20240426 09:55:17.145817 10014 rpcz_store.cc:269] 0426 09:55:12.961473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:12.961632 (+ 159us) service_pool.cc:225] Handling call
0426 09:55:14.098198 (+1136566us) spinlock_profiling.cc:243] Waited 5 ms on lock 0x7b580004ec98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce43ee 00000000003399f9 00007f9ca7b13570 00007f9ca7bd9294 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:55:17.042384 (+2944186us) spinlock_profiling.cc:243] Waited 2.89 s on lock 0x7b5400199454. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:17.042507 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6647200256}
W20240426 09:55:17.146916 10014 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 802) took 3024 ms (client timeout 1828 ms). Trace:
W20240426 09:55:17.147262 10014 rpcz_store.cc:269] 0426 09:55:14.122955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.146699 (+3023744us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.146856 (+ 157us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.147595 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.053897 11232 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:17.148241 9935 raft_consensus.cc:2454] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:17.149484 11232 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:16.869164 11239 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:17.150830 11239 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:17.152801 11239 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:17.054509 10010 rpcz_store.cc:276] 0426 09:55:12.172251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:12.350143 (+177892us) service_pool.cc:225] Handling call
0426 09:55:12.350572 (+ 429us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:12.363717 (+ 13145us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:12.363743 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:12.380512 (+ 16769us) log.cc:842] Serialized 26 byte log entry
0426 09:55:12.549018 (+168506us) raft_consensus.cc:1631] Marking committed up to 4
0426 09:55:12.549055 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:12.549121 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:15.899468 (+3350347us) spinlock_profiling.cc:243] Waited 2.1 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:55:17.053819 (+1154351us) raft_consensus.cc:1671] finished
0426 09:55:17.053843 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.054074 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12215,"raft.queue_time_us":12508,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":102,"spinlock_wait_cycles":4827008,"thread_start_us":118602,"threads_started":1,"wal-append.queue_time_us":280808}
W20240426 09:55:17.153647 10010 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 794) took 3132 ms (client timeout 2059 ms). Trace:
W20240426 09:55:17.153990 10010 rpcz_store.cc:269] 0426 09:55:14.020802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.153480 (+3132678us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.153592 (+ 112us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.154631 10083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 929) took 3109 ms. Trace:
I20240426 09:55:17.158284 10089 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 927) took 3114 ms. Trace:
I20240426 09:55:17.159145 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.160750 10083 rpcz_store.cc:276] 0426 09:55:14.045421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.118012 (+3072591us) service_pool.cc:225] Handling call
0426 09:55:17.131535 (+ 13523us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.152441 (+ 20906us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:17.152475 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:17.152521 (+ 46us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:17.152552 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.154410 (+ 1858us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b54001810d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:17.154455 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.154580 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47594240}
I20240426 09:55:17.162555 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.146924 11290 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:17.061203 11246 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:17.064865 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.055141 11236 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:17.165560 11236 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:17.061203 11171 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:17.068979 11257 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:17.077788 11079 tablet_bootstrap.cc:492] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240426 09:55:17.095486 10009 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 787) took 3098 ms (client timeout 1902 ms). Trace:
W20240426 09:55:17.171123 10009 rpcz_store.cc:269] 0426 09:55:13.997123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.095332 (+3098209us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.095432 (+ 100us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.171625 11079 tablet_bootstrap.cc:492] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
W20240426 09:55:17.171931 10009 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 799) took 3072 ms (client timeout 2028 ms). Trace:
I20240426 09:55:17.172395 11079 ts_tablet_manager.cc:1367] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 3.049s user 0.029s sys 0.000s
W20240426 09:55:17.092036 10016 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1131) took 3069 ms (client timeout 1784 ms). Trace:
W20240426 09:55:17.093189 10012 rpcz_store.cc:269] 0426 09:55:14.028790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.092648 (+3063858us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.092739 (+ 91us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.096894 10056 connection.cc:232] RPC call timeout handler was delayed by -2.3297723110000561s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1784s for a response
W20240426 09:55:17.172958 10016 rpcz_store.cc:269] 0426 09:55:14.023035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.091887 (+3068852us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.091985 (+ 98us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.095541 9905 connection.cc:232] RPC call timeout handler was delayed by -1.5031495030000315s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1902s for a response
W20240426 09:55:17.172279 10009 rpcz_store.cc:269] 0426 09:55:14.099325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.171774 (+3072449us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.171878 (+ 104us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.174408 10056 connection.cc:232] RPC call timeout handler was delayed by -2.3288723110000547s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1785s for a response
W20240426 09:55:16.999567 10011 rpcz_store.cc:269] 0426 09:55:13.034599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.034719 (+ 120us) service_pool.cc:225] Handling call
0426 09:55:16.806047 (+3771328us) spinlock_profiling.cc:243] Waited 756 ms on lock 0x7b5400056ed4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:16.806134 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1739595520}
W20240426 09:55:17.175170 10011 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1134) took 3038 ms (client timeout 2039 ms). Trace:
W20240426 09:55:17.175477 10011 rpcz_store.cc:269] 0426 09:55:14.137028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.175032 (+3038004us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.175123 (+ 91us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.095973 10087 rpcz_store.cc:269] 0426 09:55:13.920359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.922867 (+ 2508us) service_pool.cc:225] Handling call
0426 09:55:16.813598 (+2890731us) spinlock_profiling.cc:243] Waited 7.66 ms on lock 0x7b58000b0498. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:55:17.002505 (+188907us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17624448}
I20240426 09:55:17.094597 11285 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:17.102208 11054 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:17.098433 10086 rpcz_store.cc:276] 0426 09:55:14.039520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.094949 (+3055429us) service_pool.cc:225] Handling call
0426 09:55:17.095200 (+ 251us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.095352 (+ 152us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:17.095376 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:17.095413 (+ 37us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:17.095442 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.095486 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.095586 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:55:17.103580 9938 rpcz_store.cc:269] 0426 09:55:13.056868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.103101 (+4046233us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.103205 (+ 104us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.019364 10586 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:17.182371 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:17.182552 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 163242us
I20240426 09:55:17.182924 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1092) took 4127 ms. Trace:
W20240426 09:55:17.117218 9904 connection.cc:232] RPC call timeout handler was delayed by -2.6102806900000246s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2211s for a response
I20240426 09:55:17.183259 9938 rpcz_store.cc:276] 0426 09:55:13.055993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.182185 (+4126192us) service_pool.cc:225] Handling call
0426 09:55:17.182467 (+ 282us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.182618 (+ 151us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:17.182642 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:17.182680 (+ 38us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:17.182708 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.182757 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.182875 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:17.183969 11277 raft_consensus.cc:2726] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:17.183540 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.184686 10086 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:17.107579 10088 raft_consensus.cc:2454] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
W20240426 09:55:17.185294 10088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 937) took 3046 ms (client timeout 3278 ms). Trace:
I20240426 09:55:17.185458 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
W20240426 09:55:17.185634 10088 rpcz_store.cc:269] 0426 09:55:14.138959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.088761 (+2949802us) service_pool.cc:225] Handling call
0426 09:55:17.185235 (+ 96474us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:17.117511 11127 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:17.186178 10086 raft_consensus.cc:2379] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:17.186494 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.186373 11127 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:17.119519 10082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 928) took 3074 ms. Trace:
I20240426 09:55:17.186910 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:17.105185 10015 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 803) took 2982 ms (client timeout 1781 ms). Trace:
I20240426 09:55:17.119925 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1090) took 4066 ms. Trace:
I20240426 09:55:17.121331 11132 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:17.122555 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.187554 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:17.113166 11253 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:17.127269 9982 connection.cc:232] RPC call timeout handler was delayed by -2.2078971290000027s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2364s for a response
I20240426 09:55:17.187965 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.188071 9941 raft_consensus.cc:2454] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:17.188319 11253 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:17.127153 9940 rpcz_store.cc:269] 0426 09:55:13.053091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.029744 (+3976653us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.029827 (+ 83us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.188961 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:17.097303 9934 rpcz_store.cc:269] 0426 09:55:13.130945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.096730 (+3965785us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.096833 (+ 103us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.129556 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:17.191293 10081 raft_consensus.cc:2379] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:17.192019 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.193121 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.194208 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.134091 11186 leader_election.cc:288] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:17.112883 11263 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:17.196457 11263 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:17.198415 11263 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:17.124692 9933 rpcz_store.cc:269] 0426 09:55:13.052368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.055443 (+ 3075us) service_pool.cc:225] Handling call
0426 09:55:16.862344 (+3806901us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b5800004598. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:55:17.010144 (+147800us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":755760000}
W20240426 09:55:17.091024 9937 rpcz_store.cc:269] 0426 09:55:13.081918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.090437 (+4008519us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.090537 (+ 100us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.199024 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 923) took 3155 ms. Trace:
W20240426 09:55:17.125244 9939 rpcz_store.cc:269] 0426 09:55:13.052627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.068961 (+ 16334us) service_pool.cc:225] Handling call
0426 09:55:16.883240 (+3814279us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b5800004598. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:55:17.009691 (+126451us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39688448}
I20240426 09:55:17.108013 9981 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34378 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1128)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 794)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1133)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1129)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1134)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 791)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1132)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1126)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1125)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1141)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1138)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1146)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1142)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1144)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36762 (request call id 88)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1139)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1145)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 82)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 84)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 85)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1123)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1124)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1127)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1130)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 86)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1135)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1137)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1140)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 87)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1143)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1147)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1148)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1149)
I20240426 09:55:17.140712 11215 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:17.141250 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.144016 11183 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:17.144753 11118 ts_tablet_manager.cc:1398] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.111s user 0.008s sys 0.000s
I20240426 09:55:17.149771 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1091) took 4094 ms. Trace:
W20240426 09:55:17.154682 10010 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1133) took 3116 ms (client timeout 2088 ms). Trace:
I20240426 09:55:17.167356 10089 rpcz_store.cc:276] 0426 09:55:14.043940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.111937 (+3067997us) service_pool.cc:225] Handling call
0426 09:55:17.119152 (+ 7215us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.157499 (+ 38347us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:17.157531 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:17.157570 (+ 39us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:17.157598 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.158082 (+ 484us) spinlock_profiling.cc:243] Waited 38.1 ms on lock 0x7b5400125554. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:17.158123 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.158228 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":87740800}
I20240426 09:55:17.244540 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.245121 10089 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:17.246081 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:17.246189 10081 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:17.243854 9981 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34378 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1132)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1126)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1125)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1141)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1138)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 819)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 823)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 824)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1146)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 813)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1152)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1142)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1144)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36762 (request call id 88)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 811)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 805)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 810)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1139)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1145)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 82)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 84)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 85)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1123)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1124)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1127)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1130)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 86)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1135)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1137)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1140)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 87)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1143)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1147)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1148)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1149)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 806)
W20240426 09:55:17.147976 10014 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1128) took 3125 ms (client timeout 2056 ms). Trace:
I20240426 09:55:17.063594 9839 catalog_manager.cc:5484] T 6fde852ed1264555af856522b4abc093 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 1 to 3, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 3 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 09:55:17.246935 10014 rpcz_store.cc:269] 0426 09:55:14.022342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.147837 (+3125495us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.147932 (+ 95us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.247776 10014 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 786) took 3250 ms (client timeout 2243 ms). Trace:
I20240426 09:55:17.247558 9839 catalog_manager.cc:5484] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 1 to 2, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 2 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
W20240426 09:55:17.098156 10013 rpcz_store.cc:269] 0426 09:55:13.996544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.097549 (+3101005us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.097638 (+ 89us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.095824 10007 rpcz_store.cc:269] 0426 09:55:14.005227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.004135 (+2998908us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.004231 (+ 96us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.173506 9905 connection.cc:232] RPC call timeout handler was delayed by -1.1890108060000557s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2093s for a response
W20240426 09:55:17.174115 10012 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 798) took 3091 ms (client timeout 2093 ms). Trace:
W20240426 09:55:17.097633 9932 rpcz_store.cc:269] 0426 09:55:13.053535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:13.227202 (+173667us) service_pool.cc:225] Handling call
0426 09:55:16.926820 (+3699618us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b5800004598. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:55:17.097145 (+170325us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44321664}
W20240426 09:55:17.174757 10056 connection.cc:232] RPC call timeout handler was delayed by -2.2829723110000373s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1836s for a response
I20240426 09:55:17.171694 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.180078 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:17.180408 10011 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 791) took 3178 ms (client timeout 2207 ms). Trace:
W20240426 09:55:17.179975 10016 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1129) took 3157 ms (client timeout 2115 ms). Trace:
I20240426 09:55:17.184762 9938 raft_consensus.cc:1226] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:17.189956 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.190632 11253 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:17.200049 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1101) took 4081 ms. Trace:
W20240426 09:55:17.118117 10008 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1136) took 2974 ms (client timeout 1785 ms). Trace:
W20240426 09:55:17.248121 10014 rpcz_store.cc:269] 0426 09:55:13.996869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.247620 (+3250751us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.247719 (+ 99us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.200292 9904 connection.cc:232] RPC call timeout handler was delayed by -2.0774806900000158s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2803s for a response
I20240426 09:55:17.183703 10586 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:17.188094 10088 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:17.243446 9935 rpcz_store.cc:276] 0426 09:55:13.055737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.149127 (+4093390us) service_pool.cc:225] Handling call
0426 09:55:17.149380 (+ 253us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.149516 (+ 136us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:17.149539 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:17.149574 (+ 35us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:17.149600 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.149644 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.149727 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:55:17.248540 10010 rpcz_store.cc:269] 0426 09:55:14.038640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.154541 (+3115901us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.154634 (+ 93us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.261582 9904 connection.cc:232] RPC call timeout handler was delayed by -1.8612204920000295s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15200000000000002s for a response
W20240426 09:55:17.263123 10010 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1132) took 3231 ms (client timeout 2226 ms). Trace:
I20240426 09:55:17.262962 11187 consensus_queue.cc:1035] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240426 09:55:17.263504 10010 rpcz_store.cc:269] 0426 09:55:14.031676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.262961 (+3231285us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.263069 (+ 108us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.246577 10089 raft_consensus.cc:3037] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
W20240426 09:55:17.264076 10056 connection.cc:232] RPC call timeout handler was delayed by -2.2163723110000433s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.191s for a response
W20240426 09:55:17.264395 10056 connection.cc:232] RPC call timeout handler was delayed by -2.1983723110000142s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
W20240426 09:55:17.261495 10011 rpcz_store.cc:269] 0426 09:55:14.002449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.180263 (+3177814us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.180356 (+ 93us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.265122 9940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1095) took 4208 ms. Trace:
I20240426 09:55:17.247190 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
W20240426 09:55:17.261624 10016 rpcz_store.cc:269] 0426 09:55:14.022573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.179826 (+3157253us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.179922 (+ 96us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.266355 10081 raft_consensus.cc:2379] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:17.232780 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1100) took 4127 ms. Trace:
I20240426 09:55:17.263940 10086 raft_consensus.cc:2379] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
W20240426 09:55:17.187494 10015 rpcz_store.cc:269] 0426 09:55:14.123192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.105041 (+2981849us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.105134 (+ 93us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.267683 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.267601 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.238451 10090 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
W20240426 09:55:17.268337 10056 connection.cc:232] RPC call timeout handler was delayed by -2.1350971950000144s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23940000000000003s for a response
I20240426 09:55:17.200767 9936 rpcz_store.cc:276] 0426 09:55:13.054000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.113378 (+4059378us) service_pool.cc:225] Handling call
0426 09:55:17.119496 (+ 6118us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.119646 (+ 150us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:17.119667 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:17.119701 (+ 34us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:17.119745 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.119794 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.119878 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:17.178934 9979 service_pool.cc:138] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:34362 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1132)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1126)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1125)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1141)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1138)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 819)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 823)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 824)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1146)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 813)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1142)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1144)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36762 (request call id 88)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 811)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 805)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 810)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1139)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1145)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 82)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 84)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 85)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1123)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1124)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1127)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1130)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 86)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1135)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1137)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1140)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 87)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1143)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1147)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1148)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1149)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 806)
I20240426 09:55:17.242262 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1098) took 4171 ms. Trace:
W20240426 09:55:17.178926 10009 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 797) took 3110 ms (client timeout 2169 ms). Trace:
I20240426 09:55:17.199375 10085 rpcz_store.cc:276] 0426 09:55:14.043246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.108336 (+3065090us) service_pool.cc:225] Handling call
0426 09:55:17.108631 (+ 295us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:17.108912 (+ 281us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:17.108938 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:17.109328 (+ 390us) log.cc:842] Serialized 26 byte log entry
0426 09:55:17.167143 (+ 57815us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:17.167176 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.167250 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:17.198830 (+ 31580us) raft_consensus.cc:1671] finished
0426 09:55:17.198846 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.198965 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":444,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":57514,"threads_started":1,"wal-append.queue_time_us":72468}
I20240426 09:55:17.266911 9933 rpcz_store.cc:276] 0426 09:55:13.105166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.199072 (+4093906us) service_pool.cc:225] Handling call
0426 09:55:17.199348 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.199493 (+ 145us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:17.199518 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:17.199564 (+ 46us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:17.199591 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.232597 (+ 33006us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.232726 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:17.268872 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.274600 10015 raft_consensus.cc:2454] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:17.267871 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.274888 10088 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:17.269623 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1093) took 4213 ms. Trace:
I20240426 09:55:17.268931 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:17.266427 10016 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1125) took 3244 ms (client timeout 3207 ms). Trace:
W20240426 09:55:17.265394 10011 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1126) took 3243 ms (client timeout 2702 ms). Trace:
W20240426 09:55:17.267125 9904 connection.cc:232] RPC call timeout handler was delayed by -1.7919204920000311s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1597s for a response
W20240426 09:55:17.275763 10016 rpcz_store.cc:269] 0426 09:55:14.021622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.266261 (+3244639us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.266354 (+ 93us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 09:55:17.275871 10011 rpcz_store.cc:269] 0426 09:55:14.021888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.265245 (+3243357us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.265341 (+ 96us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.276939 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:17.272908 9905 connection.cc:232] RPC call timeout handler was delayed by -1.025358926000024s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22480000000000003s for a response
W20240426 09:55:17.273425 10009 rpcz_store.cc:269] 0426 09:55:14.068974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.178742 (+3109768us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.178871 (+ 129us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.278345 10083 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:17.279369 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.279600 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.280483 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.280731 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.278080 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:17.261336 10008 rpcz_store.cc:269] 0426 09:55:14.143797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.109553 (+2965756us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.112997 (+ 3444us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.262706 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
W20240426 09:55:17.271063 10056 connection.cc:232] RPC call timeout handler was delayed by -2.1002723110000261s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20390000000000003s for a response
I20240426 09:55:17.286156 10087 raft_consensus.cc:2354] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
I20240426 09:55:17.187322 10082 rpcz_store.cc:276] 0426 09:55:14.045188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.118384 (+3073196us) service_pool.cc:225] Handling call
0426 09:55:17.119051 (+ 667us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.119212 (+ 161us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:17.119238 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:17.119286 (+ 48us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:17.119321 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.119375 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.119469 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:55:17.276182 9904 connection.cc:232] RPC call timeout handler was delayed by -1.7820204920000151s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16080000000000003s for a response
I20240426 09:55:17.238482 9941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1094) took 4182 ms. Trace:
I20240426 09:55:17.277387 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.275667 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:17.282112 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.274528 9839 catalog_manager.cc:5484] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 1 to 3, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 3 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:17.283082 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.275413 9938 rpcz_store.cc:276] 0426 09:55:13.056206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.184355 (+4128149us) service_pool.cc:225] Handling call
0426 09:55:17.184618 (+ 263us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.269403 (+ 84785us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.269572 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:17.283556 10089 raft_consensus.cc:2454] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:17.284168 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.305287 10013 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
W20240426 09:55:17.275743 10012 rpcz_store.cc:269] 0426 09:55:14.082478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.173968 (+3091490us) service_pool.cc:212] Skipping call since client already timed out
0426 09:55:17.174066 (+ 98us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 09:55:17.289115 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.284835 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.307569 10011 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:17.307683 10014 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:17.282147 10085 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:17.283970 10086 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 930) took 3228 ms. Trace:
I20240426 09:55:17.309075 10086 rpcz_store.cc:276] 0426 09:55:14.055329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.283322 (+3227993us) service_pool.cc:225] Handling call
0426 09:55:17.283568 (+ 246us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.283719 (+ 151us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:17.283741 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:17.283780 (+ 39us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:17.283808 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.283853 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.283927 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:55:17.286119 10056 connection.cc:232] RPC call timeout handler was delayed by -2.0849723110000582s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2056s for a response
W20240426 09:55:17.270969 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 87334.9us
I20240426 09:55:17.265475 9940 rpcz_store.cc:276] 0426 09:55:13.056637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.189314 (+4132677us) service_pool.cc:225] Handling call
0426 09:55:17.189590 (+ 276us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:17.189873 (+ 283us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:17.189899 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:17.190272 (+ 373us) log.cc:842] Serialized 26 byte log entry
0426 09:55:17.264821 (+ 74549us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:17.264853 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.264912 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:17.264949 (+ 37us) raft_consensus.cc:1671] finished
0426 09:55:17.264959 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.265073 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9931,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":89,"thread_start_us":74258,"threads_started":1,"wal-append.queue_time_us":72740}
I20240426 09:55:17.287168 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:17.269510 11183 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20240426 09:55:17.283130 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.261255 9939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1102) took 4132 ms. Trace:
I20240426 09:55:17.238853 11124 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:17.306433 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:17.287811 9904 connection.cc:232] RPC call timeout handler was delayed by -1.6866204920000314s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1714s for a response
I20240426 09:55:17.281575 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.268438 10081 raft_consensus.cc:2454] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
W20240426 09:55:17.310122 10056 connection.cc:232] RPC call timeout handler was delayed by -2.0831723110000553s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20579999999999998s for a response
I20240426 09:55:17.311098 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.311411 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:17.312045 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.239205 11206 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:17.284705 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.288056 9941 rpcz_store.cc:276] 0426 09:55:13.056430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.188745 (+4132315us) service_pool.cc:225] Handling call
0426 09:55:17.188997 (+ 252us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.233254 (+ 44257us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:17.233284 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:17.233338 (+ 54us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:17.233366 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.238282 (+ 4916us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b54002373d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:17.238319 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.238433 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42785152}
I20240426 09:55:17.273563 11283 consensus_queue.cc:1046] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20240426 09:55:17.313027 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.638063 10012 raft_consensus.cc:2379] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
W20240426 09:55:17.638197 9904 connection.cc:232] RPC call timeout handler was delayed by -1.6812204920000227s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17200000000000001s for a response
I20240426 09:55:17.306402 10088 raft_consensus.cc:2379] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:17.288349 10083 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:17.639420 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.639603 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.410172 11054 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:17.652299 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 79) took 3510 ms. Trace:
I20240426 09:55:17.652655 10085 rpcz_store.cc:276] 0426 09:55:14.141456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.308812 (+3167356us) service_pool.cc:225] Handling call
0426 09:55:17.651860 (+343048us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.652012 (+ 152us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:17.652035 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:17.652086 (+ 51us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:17.652114 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.652163 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.652252 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:17.640897 10089 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:17.313562 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.270295 9934 rpcz_store.cc:276] 0426 09:55:13.070785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.190471 (+4119686us) service_pool.cc:225] Handling call
0426 09:55:17.190756 (+ 285us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.241387 (+ 50631us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:17.241418 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:17.241462 (+ 44us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:17.241500 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.242046 (+ 546us) spinlock_profiling.cc:243] Waited 50.4 ms on lock 0x7b54002616d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:17.242085 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.242210 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":115996928}
W20240426 09:55:17.638626 9904 connection.cc:232] RPC call timeout handler was delayed by -1.5921204920000491s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1819s for a response
I20240426 09:55:17.441419 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.642315 10016 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:17.261930 9937 rpcz_store.cc:276] 0426 09:55:13.118445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.199145 (+4080700us) service_pool.cc:225] Handling call
0426 09:55:17.199494 (+ 349us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.199699 (+ 205us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:17.199751 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:17.199794 (+ 43us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:17.199823 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.199879 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.199988 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:17.635442 11079 raft_consensus.cc:348] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:17.640655 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.591944 11127 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:17.642982 11206 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:17.643721 11124 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:17.283247 11277 raft_consensus.cc:2781] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:17.644686 10009 raft_consensus.cc:2354] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:17.645552 10013 raft_consensus.cc:3037] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 3
I20240426 09:55:17.410225 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
W20240426 09:55:17.646529 10056 connection.cc:232] RPC call timeout handler was delayed by -2.0660723110000276s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2077s for a response
W20240426 09:55:17.687021 10056 connection.cc:232] RPC call timeout handler was delayed by -2.0561723110000116s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2088s for a response
W20240426 09:55:17.687393 10056 connection.cc:232] RPC call timeout handler was delayed by -1.9319723110000382s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22260000000000002s for a response
I20240426 09:55:17.687482 10617 raft_consensus.cc:2396] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20240426 09:55:17.687708 10056 connection.cc:232] RPC call timeout handler was delayed by -1.912172311000063s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22480000000000003s for a response
I20240426 09:55:17.646973 10007 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:17.687907 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.688756 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.365167 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.689849 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.690330 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.641201 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:17.691357 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.691490 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.692029 10014 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:17.692364 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.692770 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.693329 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.693737 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:17.694401 10014 raft_consensus.cc:2379] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:17.694326 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.354625 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.368288 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.636596 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:17.317067 10008 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:17.650722 10586 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:17.648689 9979 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34390 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1156)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36762 (request call id 88)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 826)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1153)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 811)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 805)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1165)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1157)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 827)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 829)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1167)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 832)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1166)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1158)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1159)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1160)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 834)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1161)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1164)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 835)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 810)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1162)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1139)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1145)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1163)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 82)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 84)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 85)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1123)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1124)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1127)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1130)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 86)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1135)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1137)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1140)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 87)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1143)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1147)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1148)
I20240426 09:55:17.587599 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 78) took 3460 ms. Trace:
I20240426 09:55:17.659102 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:17.707237 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 56602.8us
I20240426 09:55:17.708042 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.661890 11079 raft_consensus.cc:729] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:17.707170 10087 rpcz_store.cc:276] 0426 09:55:14.126977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.287810 (+3160833us) service_pool.cc:225] Handling call
0426 09:55:17.288069 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.288229 (+ 160us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 09:55:17.314276 (+ 26047us) log.cc:842] Serialized 14 byte log entry
0426 09:55:17.587249 (+272973us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:17.587304 (+ 55us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:17.587346 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.587413 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.587541 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":28805,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":85,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":3013,"thread_start_us":272610,"threads_started":1,"wal-append.queue_time_us":327887}
I20240426 09:55:17.651964 9939 rpcz_store.cc:276] 0426 09:55:13.128739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.238729 (+4109990us) service_pool.cc:225] Handling call
0426 09:55:17.260795 (+ 22066us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.260949 (+ 154us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:17.260973 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:17.261011 (+ 38us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:17.261039 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:17.261087 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:17.261195 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:55:17.660068 9904 connection.cc:232] RPC call timeout handler was delayed by -1.5165204920000406s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19030000000000002s for a response
W20240426 09:55:17.710484 9904 connection.cc:232] RPC call timeout handler was delayed by -1.5084204920000275s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1912s for a response
W20240426 09:55:17.710837 9904 connection.cc:232] RPC call timeout handler was delayed by -1.4652204920000145s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.196s for a response
W20240426 09:55:17.711133 9904 connection.cc:232] RPC call timeout handler was delayed by -1.417520492000051s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2013s for a response
I20240426 09:55:17.665177 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:17.648677 9981 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34378 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1156)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36762 (request call id 88)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 826)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1153)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 811)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 805)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1165)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1157)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 827)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 829)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1167)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 832)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1166)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1158)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1159)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1160)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 834)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1161)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1164)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 835)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 810)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1162)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1139)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1145)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1163)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 82)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 84)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 85)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1123)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1124)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1127)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1130)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 86)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1135)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1137)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1140)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 87)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1143)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1147)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1148)
I20240426 09:55:17.666326 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.712218 10012 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:17.713012 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.713100 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.713567 10012 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:17.714358 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.481822 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.685568 11277 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:17.648677 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:17.717018 10015 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
W20240426 09:55:17.695050 10056 connection.cc:232] RPC call timeout handler was delayed by -1.7618723110000474s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24150000000000002s for a response
W20240426 09:55:17.717703 10056 connection.cc:232] RPC call timeout handler was delayed by -1.6718723110000155s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2515s for a response
W20240426 09:55:17.718093 10056 connection.cc:232] RPC call timeout handler was delayed by -1.6444960190000302s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3093s for a response
W20240426 09:55:17.718359 10056 connection.cc:232] RPC call timeout handler was delayed by -1.5035723110000276s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2702s for a response
I20240426 09:55:17.695475 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.663576 11054 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:17.482239 10612 raft_consensus.cc:1203] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 09:55:17.698089 11206 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:17.409943 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.699395 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:17.700098 11124 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:17.724026 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.636955 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:17.703197 10013 raft_consensus.cc:2454] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:17.663148 11230 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:17.708547 10010 raft_consensus.cc:2454] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:17.654328 10011 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:17.711421 9980 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:17.728226 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 77643.7us
W20240426 09:55:17.711390 9904 connection.cc:232] RPC call timeout handler was delayed by -1.3968204920000176s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2036s for a response
W20240426 09:55:17.729285 9904 connection.cc:232] RPC call timeout handler was delayed by -1.3239204920000134s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2117s for a response
W20240426 09:55:17.729583 9904 connection.cc:232] RPC call timeout handler was delayed by -1.2429204920000529s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2207s for a response
W20240426 09:55:17.729843 9904 connection.cc:232] RPC call timeout handler was delayed by -1.2141204920000064s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2239s for a response
W20240426 09:55:17.730108 9904 connection.cc:232] RPC call timeout handler was delayed by -1.2105204920000006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2243s for a response
I20240426 09:55:17.715183 10012 raft_consensus.cc:2454] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:17.689769 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.731177 10007 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:55:17.689131 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.718605 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
W20240426 09:55:17.718616 10056 connection.cc:232] RPC call timeout handler was delayed by -1.4493405950000238s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2968s for a response
W20240426 09:55:17.739823 10056 connection.cc:232] RPC call timeout handler was delayed by -1.2947723110000311s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29340000000000005s for a response
I20240426 09:55:17.736249 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:17.740211 10056 connection.cc:232] RPC call timeout handler was delayed by -1.0490723110000317s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3207s for a response
W20240426 09:55:17.740579 10056 connection.cc:232] RPC call timeout handler was delayed by -1.0184723110000391s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32410000000000005s for a response
I20240426 09:55:17.674991 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.686110 11187 raft_consensus.cc:3032] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:17.748648 10085 raft_consensus.cc:2379] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 2: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:17.748813 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.717639 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.696621 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.749872 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.750380 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.751091 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.751546 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.719676 11277 raft_consensus.cc:3037] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:17.732414 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.755563 9935 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:17.731379 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.726863 9905 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:17.756295 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.756752 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.740583 9938 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:17.757117 9935 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:17.757826 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.757956 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.758456 9938 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:17.759318 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.739884 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.747917 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:17.761304 10007 raft_consensus.cc:2379] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:17.726024 10618 raft_consensus.cc:2396] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240426 09:55:17.740942 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.769285 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.803180 9936 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:17.751714 10011 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:17.805100 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.763192 10012 raft_consensus.cc:2379] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:17.770316 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.806159 9935 raft_consensus.cc:2454] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:17.772440 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.806478 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.772604 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.807267 10012 raft_consensus.cc:2454] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:17.773161 9940 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:17.775498 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.762164 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.815259 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.776104 11206 raft_consensus.cc:2726] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:17.815841 10010 raft_consensus.cc:2454] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:17.820621 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.820736 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.793913 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.821759 10007 raft_consensus.cc:2454] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:17.822573 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.819896 10016 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:17.823546 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.823805 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.808211 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.807652 9933 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:17.817667 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.726130 10614 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:17.818322 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.748515 11187 raft_consensus.cc:729] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:17.842507 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.842722 9940 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:17.843461 10010 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:17.819008 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:17.760679 9941 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:17.817270 10585 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:17.753140 11183 raft_consensus.cc:2726] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:17.769248 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:17.846920 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.847214 9906 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:17.849431 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.851233 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:17.852018 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.853982 9904 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:17.854418 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.842507 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.878485 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.879168 10619 raft_consensus.cc:2454] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:17.880375 10617 raft_consensus.cc:1261] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 09:55:17.881201 10026 tablet_copy_service.cc:342] P 826f03b2fb9c4b9483623b041a1bf80e: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-eb5d051be8c743a18cd550e145b6564e received from {username='slave'} at 127.0.0.1:36762
I20240426 09:55:17.881342 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.881642 10026 tablet_copy_service.cc:433] P 826f03b2fb9c4b9483623b041a1bf80e: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-eb5d051be8c743a18cd550e145b6564e on tablet eb5d051be8c743a18cd550e145b6564e with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:17.882951 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.883723 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.883920 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.884354 9936 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:17.884647 9938 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:17.886436 9932 raft_consensus.cc:2454] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:17.886049 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.887987 9934 raft_consensus.cc:2379] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:17.888911 10013 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 3
I20240426 09:55:17.888598 11187 consensus_queue.cc:260] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:17.889608 11187 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:17.891050 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:17.891624 9933 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 3
I20240426 09:55:17.752480 11230 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:17.892975 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 45819.7us
I20240426 09:55:17.856316 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:17.894610 10012 raft_consensus.cc:2396] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 09:55:17.875000 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.900477 10014 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:17.865347 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.895073 10016 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:17.884305 9905 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:17.911628 9941 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:17.923843 9980 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:17.894102 9906 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:55:17.934913 10641 debug-util.cc:398] Leaking SignalData structure 0x7b08002e31c0 after lost signal to thread 8761
I20240426 09:55:17.939555 11124 consensus_queue.cc:1035] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20240426 09:55:17.940021 10641 debug-util.cc:398] Leaking SignalData structure 0x7b08002e44a0 after lost signal to thread 9873
I20240426 09:55:17.821557 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:17.941367 10011 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:55:17.941923 10083 raft_consensus.cc:2354] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
W20240426 09:55:17.954664 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 60781.8us
I20240426 09:55:17.955590 9906 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:17.975823 11054 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:17.976445 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 203124us
W20240426 09:55:17.980310 10641 debug-util.cc:398] Leaking SignalData structure 0x7b0800337ac0 after lost signal to thread 9965
I20240426 09:55:17.980808 10013 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
W20240426 09:55:17.983052 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 132655us
W20240426 09:55:17.976562 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 235513us
I20240426 09:55:17.989037 10056 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:17.989629 9904 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:17.992161 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:17.915304 11206 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:17.962131 11061 raft_consensus.cc:2781] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:17.992894 10011 raft_consensus.cc:2379] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:17.990975 11253 raft_consensus.cc:2726] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:17.993826 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:17.942965 10009 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:17.994490 11316 raft_consensus.cc:2726] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:17.941368 11127 raft_consensus.cc:2781] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:17.726308 11079 consensus_queue.cc:260] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:17.999194 9933 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:17.999536 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.985275 10015 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:17.987617 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:18.017095 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:18.007220 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.966747 11124 raft_consensus.cc:2781] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:18.008741 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:17.998656 11206 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:17.968022 11132 raft_consensus.cc:2781] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240426 09:55:17.943567 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 126382us
I20240426 09:55:18.010308 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1139) took 1140 ms. Trace:
I20240426 09:55:18.008069 9906 leader_election.cc:398] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:18.007004 11061 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:18.006209 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:18.049708 9906 leader_election.cc:401] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240426 09:55:18.017747 10016 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:18.024204 11132 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:18.018682 10015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1144) took 1009 ms. Trace:
I20240426 09:55:18.052521 10015 rpcz_store.cc:276] 0426 09:55:17.009581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.648454 (+638873us) service_pool.cc:225] Handling call
0426 09:55:18.018624 (+370170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:18.020408 10007 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:18.008813 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:18.020984 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:18.053808 10010 raft_consensus.cc:2354] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:18.022311 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:18.015856 11127 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:18.024152 9825 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.763s user 0.021s sys 0.040s
I20240426 09:55:18.015413 11253 raft_consensus.cc:2726] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:18.017627 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:18.030984 10011 rpcz_store.cc:276] 0426 09:55:16.867540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.959901 (+1092361us) service_pool.cc:225] Handling call
0426 09:55:17.992022 (+ 32121us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b580004ec98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:55:18.008400 (+ 16378us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":62978816}
I20240426 09:55:18.062537 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1123) took 4050 ms. Trace:
W20240426 09:55:18.021696 10641 debug-util.cc:398] Leaking SignalData structure 0x7b08003416a0 after lost signal to thread 10040
I20240426 09:55:18.020195 11124 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:18.024106 11206 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:18.031339 9941 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
W20240426 09:55:18.009214 10084 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.749s user 0.009s sys 0.030s
W20240426 09:55:18.050596 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 76108.4us
I20240426 09:55:18.050033 10014 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:18.055526 10013 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:18.070115 10014 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1127) took 4048 ms. Trace:
W20240426 09:55:18.055068 9825 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.794s user 0.021s sys 0.040s
I20240426 09:55:18.059484 10012 raft_consensus.cc:2379] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
W20240426 09:55:18.058712 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 284138us
I20240426 09:55:18.070451 10014 rpcz_store.cc:276] 0426 09:55:14.022098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:18.069436 (+4047338us) service_pool.cc:225] Handling call
0426 09:55:18.069695 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:18.069829 (+ 134us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:18.069850 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:18.069883 (+ 33us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:18.069932 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:18.069978 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:18.070068 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:18.071960 10014 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1130) took 4049 ms. Trace:
I20240426 09:55:18.072253 10014 rpcz_store.cc:276] 0426 09:55:14.022809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:18.071329 (+4048520us) service_pool.cc:225] Handling call
0426 09:55:18.071570 (+ 241us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:18.071704 (+ 134us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:18.071728 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:18.071762 (+ 34us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:18.071789 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:18.071836 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:18.071918 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:18.074229 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:18.075258 10014 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 86) took 3968 ms. Trace:
I20240426 09:55:18.062889 10010 rpcz_store.cc:276] 0426 09:55:14.012152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:18.061879 (+4049727us) service_pool.cc:225] Handling call
0426 09:55:18.062146 (+ 267us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:18.062283 (+ 137us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:18.062305 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:18.062339 (+ 34us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:18.062365 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:18.062410 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:18.062491 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:18.076138 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:18.076218 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:18.076771 10012 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:18.064589 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 85) took 4082 ms. Trace:
I20240426 09:55:18.065786 10015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 84) took 4114 ms. Trace:
W20240426 09:55:18.077924 9825 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 896731us
I20240426 09:55:18.077749 10015 rpcz_store.cc:276] 0426 09:55:13.950951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:18.065107 (+4114156us) service_pool.cc:225] Handling call
0426 09:55:18.065372 (+ 265us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:18.065518 (+ 146us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:18.065541 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:18.065578 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:18.065606 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:18.065651 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:18.065738 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:18.078985 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:18.079566 10088 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:18.079681 9906 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:18.080391 11253 raft_consensus.cc:2781] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:18.080775 11253 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:18.081125 11253 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:18.081859 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:18.083916 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:18.085079 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:18.086731 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 236801us
I20240426 09:55:18.049887 11061 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:18.089040 11253 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:18.075177 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:18.091248 11253 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:18.079347 10014 rpcz_store.cc:276] 0426 09:55:14.106700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:18.073004 (+3966304us) service_pool.cc:225] Handling call
0426 09:55:18.073227 (+ 223us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:18.074979 (+ 1752us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:18.075009 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:18.075047 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:18.075077 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:18.075125 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:18.075214 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:55:18.093812 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 104602us
W20240426 09:55:18.066165 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 77965.3us
I20240426 09:55:18.077252 10007 rpcz_store.cc:276] 0426 09:55:13.982424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:18.063893 (+4081469us) service_pool.cc:225] Handling call
0426 09:55:18.064175 (+ 282us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:18.064316 (+ 141us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:18.064340 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:18.064377 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:18.064405 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:18.064453 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:18.064539 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:18.096015 9904 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:18.097039 9904 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:18.098416 11253 raft_consensus.cc:2781] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:18.098804 11253 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:18.099169 11253 raft_consensus.cc:3037] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:18.060732 10016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 82) took 4243 ms. Trace:
I20240426 09:55:18.100828 10016 rpcz_store.cc:276] 0426 09:55:13.817200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:18.059981 (+4242781us) service_pool.cc:225] Handling call
0426 09:55:18.060299 (+ 318us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:18.060458 (+ 159us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:18.060482 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:18.060520 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:18.060548 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:18.060593 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:18.060683 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:18.077603 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:18.102387 9904 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:18.103516 10012 raft_consensus.cc:2379] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:18.104022 9979 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:18.106043 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:18.106894 10087 raft_consensus.cc:2379] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:18.107309 11062 raft_consensus.cc:2781] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:18.107728 11062 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:18.108042 11062 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:18.108702 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:18.110203 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:18.110356 9904 leader_election.cc:398] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:18.110898 10087 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:18.102504 10016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 87) took 1166 ms. Trace:
I20240426 09:55:18.112011 10016 rpcz_store.cc:276] 0426 09:55:16.936353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:18.101787 (+1165434us) service_pool.cc:225] Handling call
0426 09:55:18.102073 (+ 286us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:18.102223 (+ 150us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:18.102247 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:18.102286 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:18.102313 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:18.102364 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:18.102458 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:18.113111 9904 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:18.116580 11062 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:18.119164 11062 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:18.119347 10013 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:18.120250 10013 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1145) took 1110 ms. Trace:
I20240426 09:55:18.122046 9979 leader_election.cc:398] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
I20240426 09:55:18.104727 10012 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1143) took 1095 ms. Trace:
I20240426 09:55:18.054705 11230 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:18.087268 9906 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:18.115008 10016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1147) took 1104 ms. Trace:
I20240426 09:55:18.051716 11316 raft_consensus.cc:2726] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:18.128082 10013 rpcz_store.cc:276] 0426 09:55:17.009822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:18.008586 (+998764us) service_pool.cc:225] Handling call
0426 09:55:18.120171 (+111585us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:18.059763 11132 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:18.124488 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:18.157248 10012 rpcz_store.cc:276] 0426 09:55:17.009324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:18.104080 (+1094756us) service_pool.cc:225] Handling call
0426 09:55:18.104323 (+ 243us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:18.104455 (+ 132us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:18.104477 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:18.104513 (+ 36us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:18.104541 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:18.104589 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:18.104676 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:18.122758 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:18.216444 10016 rpcz_store.cc:276] 0426 09:55:17.010318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:18.113911 (+1103593us) service_pool.cc:225] Handling call
0426 09:55:18.114328 (+ 417us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:18.114558 (+ 230us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:18.114605 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:18.114669 (+ 64us) raft_consensus.cc:1631] Marking committed up to 4
0426 09:55:18.114698 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:18.114792 (+ 94us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:18.114934 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:55:18.073454 10084 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.813s user 0.009s sys 0.031s
I20240426 09:55:18.160045 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1148) took 1149 ms. Trace:
I20240426 09:55:18.220777 10010 rpcz_store.cc:276] 0426 09:55:17.010597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:18.123745 (+1113148us) service_pool.cc:225] Handling call
0426 09:55:18.159519 (+ 35774us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:18.159703 (+ 184us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:18.159750 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:18.159788 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:18.159816 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:18.159884 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:18.159994 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:18.073624 11124 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
W20240426 09:55:18.067987 10641 debug-util.cc:398] Leaking SignalData structure 0x7b08002f33e0 after lost signal to thread 10041
I20240426 09:55:18.059651 11127 raft_consensus.cc:3037] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:18.117424 11061 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:18.156554 11315 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240426 09:55:18.213826 10585 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletCopyService.EndTabletCopySession -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 55280.5us
I20240426 09:55:18.244566 11315 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:18.244946 11315 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:18.244946 10585 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:18.243796 10641 debug-util.cc:398] Leaking SignalData structure 0x7b0800330380 after lost signal to thread 10114
I20240426 09:55:18.222748 11316 raft_consensus.cc:2726] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:18.257607 11328 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:18.257912 11316 raft_consensus.cc:2781] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:18.258380 11316 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:18.258706 11328 raft_consensus.cc:686] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:18.258900 11316 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
W20240426 09:55:18.258096 10641 debug-util.cc:398] Leaking SignalData structure 0x7b080034d380 after lost signal to thread 10644
I20240426 09:55:18.262697 11318 raft_consensus.cc:2781] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:18.258598 11320 consensus_queue.cc:1035] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
W20240426 09:55:18.281774 11307 log.cc:924] Time spent T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.158s user 0.000s sys 0.000s
I20240426 09:55:18.274567 11318 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:18.272445 11277 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:18.295140 11315 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:18.296159 11277 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:18.285863 11253 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:18.291507 11303 log.cc:924] Time spent T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.242s user 0.003s sys 0.000s
I20240426 09:55:18.285900 11318 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:18.276721 11328 consensus_queue.cc:237] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:18.309010 11316 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:18.311530 11316 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:18.158989 9904 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:18.448413 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 289321us
I20240426 09:55:18.449741 9904 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:18.214222 9979 leader_election.cc:401] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20240426 09:55:18.451406 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 329362us
W20240426 09:55:18.342893 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 255482us
I20240426 09:55:18.350142 10013 raft_consensus.cc:1226] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:18.222559 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:18.456853 11187 raft_consensus.cc:2726] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
W20240426 09:55:18.266471 10641 debug-util.cc:398] Leaking SignalData structure 0x7b080034d0c0 after lost signal to thread 10645
I20240426 09:55:18.459757 9904 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:18.310160 11318 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:18.373106 10008 consensus_meta.cc:287] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.656s user 0.009s sys 0.006s
I20240426 09:55:18.461941 10008 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:18.462698 11236 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:18.365502 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 109985us
I20240426 09:55:18.463991 11318 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:18.463531 11236 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:18.379694 11320 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:18.465274 11320 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:18.465731 11320 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:18.467155 10011 raft_consensus.cc:1226] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:18.467643 10613 raft_consensus.cc:1226] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:18.385738 11079 ts_tablet_manager.cc:1398] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 1.211s user 0.054s sys 0.055s
W20240426 09:55:18.471005 11079 ts_tablet_manager.cc:1430] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: Tablet startup took 1297ms
I20240426 09:55:18.463047 11331 raft_consensus.cc:2726] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:18.388960 10620 raft_consensus.cc:1226] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:18.401961 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 804) took 1231 ms. Trace:
W20240426 09:55:18.408604 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 251195us
I20240426 09:55:18.409636 10618 raft_consensus.cc:1203] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 09:55:18.417248 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:18.474602 10009 rpcz_store.cc:276] 0426 09:55:17.170467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.688894 (+518427us) service_pool.cc:225] Handling call
0426 09:55:18.401394 (+712500us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b540027e2d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:18.401638 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":334382848}
I20240426 09:55:18.424243 10615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45564 (request call id 175) took 1068 ms. Trace:
I20240426 09:55:18.462719 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 813) took 1287 ms. Trace:
I20240426 09:55:18.425104 11037 raft_consensus.cc:2781] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:18.410705 11061 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:18.480578 9937 raft_consensus.cc:3037] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:18.427222 11253 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:18.481757 11328 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:18.482199 11328 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:18.482654 11328 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:18.436167 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:18.443305 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:18.486913 11316 raft_consensus.cc:2726] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:18.487782 9904 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:18.488338 11206 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:18.488967 9904 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:18.489940 11253 raft_consensus.cc:2781] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:18.490367 11253 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:18.490854 11253 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:18.477963 10615 rpcz_store.cc:276] 0426 09:55:17.355256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.365520 (+ 10264us) service_pool.cc:225] Handling call
0426 09:55:17.445164 (+ 79644us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.445409 (+ 245us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 09:55:17.722953 (+277544us) log.cc:842] Serialized 14 byte log entry
0426 09:55:18.422208 (+699255us) spinlock_profiling.cc:243] Waited 3.58 ms on lock 0x7b180034ea60. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9ca28d56ac 00007f9c9fcfe20c 00007f9ca8b37789 00007f9c9fd1b0eb 00007f9c9fd194ef 00007f9c9fd1b652 00007f9ca28c743b 00007f9ca28ccee3 00007f9ca28cd3fb
0426 09:55:18.423534 (+ 1326us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:18.423619 (+ 85us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:18.423699 (+ 80us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:18.423801 (+ 102us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:18.424036 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":516343,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":388,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":258,"spinlock_wait_cycles":8240000,"thread_start_us":553379,"threads_started":2,"wal-append.queue_time_us":327004}
I20240426 09:55:18.492650 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:18.491433 11206 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:18.442409 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1140) took 1574 ms. Trace:
I20240426 09:55:18.496213 10007 rpcz_store.cc:276] 0426 09:55:16.868824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:18.096169 (+1227345us) service_pool.cc:225] Handling call
0426 09:55:18.410966 (+314797us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:18.411422 (+ 456us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:18.411443 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:18.411500 (+ 57us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:18.411528 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:18.411609 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:18.441998 (+ 30389us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b540027e2d0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c45b99 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:18.442147 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":713593728}
I20240426 09:55:18.452896 9979 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:18.453823 9906 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:18.479113 10008 rpcz_store.cc:276] 0426 09:55:17.175736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.286920 (+111184us) service_pool.cc:225] Handling call
0426 09:55:18.462624 (+1175704us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:18.495245 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:18.495740 11316 raft_consensus.cc:2781] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:18.298265 11315 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:18.490164 10055 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
W20240426 09:55:18.501693 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 48011.8us
I20240426 09:55:18.501860 11331 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:18.502318 11331 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:18.502722 11331 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
W20240426 09:55:18.471405 11079 ts_tablet_manager.cc:1432] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: Trace:
0426 09:55:09.323242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:09.323380 (+ 138us) service_pool.cc:225] Handling call
0426 09:55:09.459479 (+136099us) ts_tablet_manager.cc:903] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
0426 09:55:11.024144 (+1564665us) tablet_copy_client.cc:353] Tablet copy session begun
0426 09:55:11.122268 (+ 98124us) tablet_metadata.cc:641] Metadata flushed
0426 09:55:11.122480 (+ 212us) tablet_copy_client.cc:482] Wrote new tablet metadata
0426 09:55:11.630874 (+508394us) tablet_copy_client.cc:403] Wrote new consensus metadata
0426 09:55:11.667209 (+ 36335us) tablet_replica.cc:205] Creating consensus instance
0426 09:55:11.667567 (+ 358us) ts_tablet_manager.cc:958] Replica registered.
0426 09:55:11.667625 (+ 58us) inbound_call.cc:173] Queueing success response
0426 09:55:14.123527 (+2455902us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0426 09:55:17.174728 (+3051201us) ts_tablet_manager.cc:1399] Starting tablet replica
0426 09:55:17.286876 (+112148us) tablet_replica.cc:259] Starting instrumentation
0426 09:55:17.287471 (+ 595us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":308,"mutex_wait_us":36144,"raft.queue_time_us":201157,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":10372,"tablet-copy.queue_time_us":113456,"tablet-download-pool-.queue_time_us":73379,"tablet-download-pool-.run_cpu_time_us":1405,"tablet-download-pool-.run_wall_time_us":2114651,"thread_start_us":1377874,"threads_started":6}
I20240426 09:55:18.503582 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1124) took 4489 ms. Trace:
I20240426 09:55:18.503981 10011 rpcz_store.cc:276] 0426 09:55:14.014148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:18.062272 (+4048124us) service_pool.cc:225] Handling call
0426 09:55:18.466825 (+404553us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:18.467603 (+ 778us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:18.503332 (+ 35729us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b540020fdd0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c45b99 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:18.503499 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":929549696}
I20240426 09:55:18.501031 11316 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:18.507572 11316 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:18.509253 11037 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:18.509745 11037 raft_consensus.cc:3037] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:18.512432 11062 raft_consensus.cc:1050] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:18.508006 10015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1135) took 4336 ms. Trace:
I20240426 09:55:18.507506 10620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45564 (request call id 171) took 1196 ms. Trace:
I20240426 09:55:18.502544 9906 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:55:18.971585 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 469141us
I20240426 09:55:18.972328 9906 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:55:18.514626 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 61926.4us
I20240426 09:55:18.975734 9906 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:18.552755 10084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 922) took 4512 ms. Trace:
W20240426 09:55:18.537673 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 58761.4us
I20240426 09:55:18.520541 10015 rpcz_store.cc:276] 0426 09:55:14.139973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:18.078789 (+3938816us) service_pool.cc:225] Handling call
0426 09:55:18.079029 (+ 240us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:18.079194 (+ 165us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 09:55:18.347495 (+268301us) log.cc:842] Serialized 14 byte log entry
0426 09:55:18.368672 (+ 21177us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:18.395059 (+ 26387us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:18.452161 (+ 57102us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:18.455354 (+ 3193us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:18.475965 (+ 20611us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":324079,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":42,"raft.queue_time_us":24109,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":1072,"thread_start_us":151715,"threads_started":2,"wal-append.queue_time_us":74698}
W20240426 09:55:18.781591 9982 leader_election.cc:334] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Timed out: RequestConsensusVote RPC to 127.8.141.129:38113 timed out after 1.591s (SENT)
I20240426 09:55:18.519907 10620 rpcz_store.cc:276] 0426 09:55:17.310849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.311585 (+ 736us) service_pool.cc:225] Handling call
0426 09:55:17.633911 (+322326us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:18.478038 (+844127us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:18.507282 (+ 29244us) spinlock_profiling.cc:243] Waited 729 ms on lock 0x7b5400451854. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:18.507453 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1676863872}
I20240426 09:55:18.803468 11037 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:18.988982 11037 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:18.515651 11061 raft_consensus.cc:2781] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:18.990478 11061 raft_consensus.cc:686] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:18.531466 11062 consensus_queue.cc:237] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:18.546341 11124 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.036943 11124 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:19.037484 11124 raft_consensus.cc:2726] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:18.558676 11127 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:19.039835 11127 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:18.541146 11132 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:19.041981 11132 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:18.808310 11236 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:18.813076 11316 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.046506 11316 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:18.547801 11320 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.075210 11320 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:18.541379 11253 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.079289 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:18.559208 11328 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.084035 11328 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:18.800531 11331 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.086493 11253 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:19.087425 11331 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:18.902478 9904 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:19.089274 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 187003us
W20240426 09:55:18.925681 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 435667us
I20240426 09:55:18.947121 9839 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58806 (request call id 83) took 7406 ms. Trace:
I20240426 09:55:18.974409 11350 raft_consensus.cc:2781] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:18.985059 9982 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:18.991653 11351 raft_consensus.cc:2741] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20240426 09:55:18.991873 9906 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:18.991786 11061 consensus_queue.cc:237] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:18.980661 10084 rpcz_store.cc:276] 0426 09:55:14.040079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.104051 (+3063972us) service_pool.cc:225] Handling call
0426 09:55:17.104363 (+ 312us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:17.104559 (+ 196us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 09:55:17.260266 (+155707us) log.cc:842] Serialized 14 byte log entry
0426 09:55:18.498608 (+1238342us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:18.518581 (+ 19973us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:18.518719 (+ 138us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:18.552341 (+ 33622us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:18.552562 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13342,"prepare.queue_time_us":142293,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"raft.queue_time_us":23381,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":6108,"thread_start_us":1276766,"threads_started":3,"wal-append.queue_time_us":798603}
I20240426 09:55:18.997920 9979 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:19.020068 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.023041 11337 consensus_queue.cc:1046] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20240426 09:55:19.024859 9937 consensus_meta.cc:287] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.543s user 0.010s sys 0.001s
I20240426 09:55:19.044924 10614 raft_consensus.cc:1226] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20240426 09:55:19.048508 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.049872 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:19.057263 11336 raft_consensus.cc:2781] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240426 09:55:19.067250 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 82549.5us
I20240426 09:55:19.068964 11315 consensus_queue.cc:1046] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
W20240426 09:55:19.070973 10014 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.004s sys 0.008s
W20240426 09:55:19.114639 10014 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s user 0.005s sys 0.008s
I20240426 09:55:19.090692 9904 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:19.094202 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.118978 10009 raft_consensus.cc:2379] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:19.120657 11331 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:19.121603 11315 raft_consensus.cc:3032] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:19.122022 11315 raft_consensus.cc:729] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:19.114555 11350 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:19.123045 10614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51158 (request call id 128) took 1149 ms. Trace:
I20240426 09:55:19.122911 11315 consensus_queue.cc:260] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:19.124246 11315 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:19.091970 10613 raft_consensus.cc:1261] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240426 09:55:19.127074 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 135312us
I20240426 09:55:19.126662 11350 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:19.142659 11354 raft_consensus.cc:1050] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:19.128063 11337 raft_consensus.cc:3032] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:19.129554 10055 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:19.114689 11336 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:19.115512 9937 raft_consensus.cc:2454] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:55:19.127745 9940 raft_consensus.cc:3037] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.131172 9904 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:19.132069 11331 raft_consensus.cc:2781] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:19.132602 10585 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:19.133178 11230 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:19.133615 11187 raft_consensus.cc:2726] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 09:55:19.133822 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 352447us
I20240426 09:55:19.134956 11236 consensus_queue.cc:1035] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:19.116341 9839 rpcz_store.cc:276] 0426 09:55:11.540295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:11.540425 (+ 130us) service_pool.cc:225] Handling call
0426 09:55:18.946901 (+7406476us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:17.942427 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:17.960385 (+ 17958us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:17.960425 (+ 40us) write_op.cc:427] Acquired schema lock
0426 09:55:17.960444 (+ 19us) tablet.cc:660] Decoding operations
0426 09:55:17.964134 (+ 3690us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:17.964209 (+ 75us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:17.964238 (+ 29us) tablet.cc:683] Acquiring locks for 3 operations
0426 09:55:17.964570 (+ 332us) tablet.cc:699] Row locks acquired
0426 09:55:17.964588 (+ 18us) write_op.cc:260] PREPARE: finished
0426 09:55:17.964731 (+ 143us) write_op.cc:270] Start()
0426 09:55:17.964839 (+ 108us) write_op.cc:276] Timestamp: P: 1714125317964690 usec, L: 0
0426 09:55:17.964859 (+ 20us) op_driver.cc:331] REPLICATION: starting
0426 09:55:18.016730 (+ 51871us) log.cc:842] Serialized 1297 byte log entry
0426 09:55:18.314222 (+297492us) op_driver.cc:444] REPLICATION: finished
0426 09:55:18.451452 (+137230us) write_op.cc:301] APPLY: starting
0426 09:55:18.451560 (+ 108us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:18.451725 (+ 165us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:18.452475 (+ 750us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:18.453853 (+ 1378us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:18.454118 (+ 265us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0426 09:55:18.454177 (+ 59us) write_op.cc:312] APPLY: finished
0426 09:55:18.521186 (+ 67009us) log.cc:842] Serialized 34 byte log entry
0426 09:55:18.523394 (+ 2208us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:18.523589 (+ 195us) write_op.cc:446] Released schema lock
0426 09:55:18.523641 (+ 52us) write_op.cc:341] FINISH: Updating metrics
0426 09:55:18.524007 (+ 366us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:18.524044 (+ 37us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":136699,"apply.run_cpu_time_us":3882,"apply.run_wall_time_us":72972,"num_ops":3,"prepare.queue_time_us":628111,"prepare.run_cpu_time_us":48310,"prepare.run_wall_time_us":291943,"raft.queue_time_us":32201,"raft.run_cpu_time_us":2578,"raft.run_wall_time_us":65226,"replication_time_us":349275,"thread_start_us":669381,"threads_started":4,"wal-append.queue_time_us":237404}]]}
I20240426 09:55:19.149426 11354 consensus_queue.cc:237] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:19.145414 11336 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.143291 11337 raft_consensus.cc:729] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:19.154176 11337 consensus_queue.cc:260] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:19.155514 11337 raft_consensus.cc:3037] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.156687 10586 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:19.147529 11230 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:19.159186 11230 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:19.160279 9940 raft_consensus.cc:2454] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:55:19.160776 11351 raft_consensus.cc:2726] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:19.161132 11132 raft_consensus.cc:2781] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:19.161651 11132 raft_consensus.cc:686] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:19.162539 11132 consensus_queue.cc:237] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:19.163691 9982 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.141.129:38113 timed out after 1.596s (SENT) [suppressed 94 similar messages]
W20240426 09:55:19.165158 9982 leader_election.cc:334] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Timed out: RequestConsensusVote RPC to 127.8.141.129:38113 timed out after 1.663s (SENT)
I20240426 09:55:19.144968 11331 raft_consensus.cc:686] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:19.166944 9982 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:19.125514 10614 rpcz_store.cc:276] 0426 09:55:17.973547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:17.974655 (+ 1108us) service_pool.cc:225] Handling call
0426 09:55:17.975078 (+ 423us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:19.120827 (+1145749us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:19.122721 (+ 1894us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54005528d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:19.122984 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2459573504}
I20240426 09:55:19.145325 10055 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:19.132539 10014 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1137) took 4984 ms. Trace:
I20240426 09:55:19.173880 10586 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:19.173722 11331 consensus_queue.cc:237] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.174927 11037 raft_consensus.cc:2781] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:19.175185 9904 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:19.171226 10014 rpcz_store.cc:276] 0426 09:55:14.148140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:18.092957 (+3944817us) service_pool.cc:225] Handling call
0426 09:55:18.093231 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:18.093399 (+ 168us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 09:55:18.498282 (+404883us) log.cc:842] Serialized 14 byte log entry
0426 09:55:19.132133 (+633851us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:19.132204 (+ 71us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:19.132270 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:19.132352 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:19.132462 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":136542,"prepare.queue_time_us":310257,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":42,"raft.queue_time_us":16810,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"spinlock_wait_cycles":13056,"thread_start_us":866025,"threads_started":2,"wal-append.queue_time_us":298606}
I20240426 09:55:19.179103 11187 raft_consensus.cc:2726] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:19.179276 11333 raft_consensus.cc:2781] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:19.179582 11343 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:19.182853 11352 raft_consensus.cc:2726] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:19.180267 11328 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:19.183779 11328 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:19.184160 11328 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:19.186790 11253 raft_consensus.cc:2781] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:19.187240 11253 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:19.187614 11253 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.180171 11037 raft_consensus.cc:686] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:19.191814 11037 consensus_queue.cc:237] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:19.181545 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 49043.9us
I20240426 09:55:19.180040 11343 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:19.195739 11343 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:19.199002 11351 consensus_queue.cc:1046] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 09:55:19.200004 11320 raft_consensus.cc:3032] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:19.200392 11320 raft_consensus.cc:729] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:19.200865 10055 leader_election.cc:302] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:19.201315 11320 consensus_queue.cc:260] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:19.202672 11320 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:19.208916 11350 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.211524 9904 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:19.213147 9904 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:19.213783 11351 raft_consensus.cc:2726] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:19.216231 11350 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:19.244961 11333 raft_consensus.cc:686] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:19.241012 11350 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:19.247256 11350 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:19.247635 11350 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:55:19.249002 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 252039us
I20240426 09:55:19.250481 9979 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:19.243126 11230 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.239784 11336 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.245625 9838 catalog_manager.cc:5484] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 2 to 3, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 3 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:19.252288 11236 raft_consensus.cc:2781] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:19.290441 11236 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:19.290884 11236 raft_consensus.cc:3037] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.256374 9904 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:19.258482 11253 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:19.284088 11343 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.294275 11253 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:19.260316 11328 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.246111 11333 consensus_queue.cc:237] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.268695 11230 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:19.269124 9906 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:19.269857 11358 raft_consensus.cc:2781] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:19.277029 10009 raft_consensus.cc:1226] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:19.280409 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.281052 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.281605 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.283921 11350 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.294867 11253 raft_consensus.cc:2726] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:19.291786 11336 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:19.301007 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 54418.1us
I20240426 09:55:19.302022 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:19.303880 10014 raft_consensus.cc:2354] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:19.304313 10008 raft_consensus.cc:1261] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:19.305189 11350 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:19.305763 9904 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:19.306682 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 105964us
I20240426 09:55:19.307013 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.307397 11343 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:19.307971 10055 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:19.308239 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.309361 11253 raft_consensus.cc:2781] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:19.309753 11253 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:19.310127 11253 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:19.310201 11328 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:19.310448 9904 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:19.310832 11328 raft_consensus.cc:2726] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:19.311893 9838 catalog_manager.cc:5484] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 2, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 2 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:19.313014 11331 raft_consensus.cc:2726] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:19.313333 9838 catalog_manager.cc:5484] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 2, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 2 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:19.314342 9904 leader_election.cc:398] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:19.314689 9904 leader_election.cc:401] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 09:55:19.315272 11331 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.301435 11358 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:19.316944 11358 raft_consensus.cc:3037] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.317307 11253 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.319171 11253 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:19.320235 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.321794 11336 raft_consensus.cc:2781] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:19.322211 11336 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:19.322796 11336 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:19.324057 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:19.325349 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:19.327116 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.328086 10008 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:19.337862 9906 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:19.338851 11253 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election won for term 2
W20240426 09:55:19.339154 9982 leader_election.cc:334] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Timed out: RequestConsensusVote RPC to 127.8.141.129:38113 timed out after 1.692s (SENT)
I20240426 09:55:19.339466 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:19.339680 11253 raft_consensus.cc:686] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:19.341590 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.342854 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.343425 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.340837 11253 consensus_queue.cc:237] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.344507 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.345685 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.336104 9838 catalog_manager.cc:5484] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:19.349972 11315 consensus_queue.cc:1046] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:19.327049 11331 raft_consensus.cc:2726] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:19.353221 9838 catalog_manager.cc:5484] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 2, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 2 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:19.354950 11336 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:19.355371 10008 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
W20240426 09:55:19.356477 9982 leader_election.cc:334] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Timed out: RequestConsensusVote RPC to 127.8.141.129:38113 timed out after 1.921s (SENT)
I20240426 09:55:19.357066 11336 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:19.355046 9838 catalog_manager.cc:5484] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 2, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 2 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:19.351063 11343 raft_consensus.cc:3032] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:19.361188 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:19.362639 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:19.364044 10055 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:19.364060 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:19.364964 10090 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:19.365598 10055 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:19.366492 11361 raft_consensus.cc:2781] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:19.366914 11361 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:19.367254 11361 raft_consensus.cc:3037] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.368636 11358 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:19.362663 9906 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:19.361102 11343 raft_consensus.cc:729] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:19.372525 9838 catalog_manager.cc:5484] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 2, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 2 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:19.374918 9838 catalog_manager.cc:5484] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 2, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 2 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
W20240426 09:55:19.377316 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 47211.7us
I20240426 09:55:19.379701 10620 raft_consensus.cc:1261] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 1 LEARNER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:19.372388 11336 raft_consensus.cc:2781] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:19.380102 11331 raft_consensus.cc:2726] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:19.380714 11336 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:19.384141 11336 raft_consensus.cc:3037] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.384792 11361 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:19.383175 9906 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:19.387632 9906 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:19.376359 9838 catalog_manager.cc:5484] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 2, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 2 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:19.388422 9906 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:19.389572 9906 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:19.389163 9838 catalog_manager.cc:5484] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index -1 to 2, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:19.311921 11236 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:19.393476 11253 raft_consensus.cc:2726] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:19.394831 9838 catalog_manager.cc:5484] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:19.396255 11343 consensus_queue.cc:260] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.393824 11350 raft_consensus.cc:2781] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:19.399297 11331 raft_consensus.cc:2781] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240426 09:55:19.400131 9982 leader_election.cc:334] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Timed out: RequestConsensusVote RPC to 127.8.141.129:38113 timed out after 2.064s (SENT)
I20240426 09:55:19.401100 9838 catalog_manager.cc:5484] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 2, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 2 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:19.393133 11361 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:19.401716 11320 raft_consensus.cc:2726] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:19.401300 11350 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:19.413069 11350 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:19.413589 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:19.414170 10090 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:19.415076 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:19.415855 10081 raft_consensus.cc:2379] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:19.417100 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.401950 11331 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:19.419847 9906 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:19.399497 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:19.405426 11343 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:19.423377 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:19.424196 10084 raft_consensus.cc:2379] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 2: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:19.407132 11333 consensus_queue.cc:1035] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:19.410943 11358 leader_election.cc:288] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:19.411918 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:19.412151 10055 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:19.420759 11320 raft_consensus.cc:2781] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:19.420287 11331 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:19.433910 9906 leader_election.cc:398] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:19.433353 11320 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:19.435138 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:19.436070 10081 raft_consensus.cc:2379] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:19.435969 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.437077 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.437652 10086 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.437327 10088 raft_consensus.cc:2354] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:19.439208 10055 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:19.440832 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.441134 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.441382 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.441761 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.442129 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.442360 10087 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:19.442647 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.443271 10089 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.443854 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.444523 9906 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:19.444949 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:19.435261 11320 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:19.445307 11253 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:19.443187 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.445784 11253 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:19.446202 11253 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.443895 11331 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.445600 10084 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:19.448799 11331 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:19.440748 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.448808 11358 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:19.452031 11358 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:19.452335 11358 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.445901 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.445998 9906 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:19.463800 10086 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:19.439043 11336 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:19.468329 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.472262 11336 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:19.441531 10082 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:19.545136 11253 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:19.548064 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 72537.1us
I20240426 09:55:19.547981 10008 raft_consensus.cc:1226] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:19.553860 11320 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.537514 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.556037 11320 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:19.561239 10089 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:19.535324 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.544392 11331 raft_consensus.cc:3037] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.565104 9937 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:19.562882 11236 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:19.563290 11336 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:19.565986 11336 raft_consensus.cc:686] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:19.567111 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:19.567781 11253 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:19.561416 11358 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:19.539063 9979 leader_election.cc:334] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 1.690s (SENT)
I20240426 09:55:19.539563 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.569597 10087 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
W20240426 09:55:19.572333 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 126410us
I20240426 09:55:19.573869 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:19.566901 11336 consensus_queue.cc:237] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.579217 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.581866 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.584156 9937 raft_consensus.cc:2379] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.584475 9941 raft_consensus.cc:1226] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:19.578047 10081 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:19.588745 10055 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:19.589785 11354 raft_consensus.cc:2781] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:19.590150 11354 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:19.590482 11354 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.590724 10055 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:19.591131 9906 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:19.594877 11230 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:19.595124 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.595937 9941 raft_consensus.cc:2379] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:19.595927 11230 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:19.596963 11037 consensus_queue.cc:1046] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:19.597975 11230 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:19.598980 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:19.599969 10620 raft_consensus.cc:2379] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 4: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:19.600723 11331 raft_consensus.cc:2726] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:19.570398 11350 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.601341 9934 raft_consensus.cc:2454] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:19.601989 11352 raft_consensus.cc:3032] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:19.602335 11352 raft_consensus.cc:729] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:19.602567 11350 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:19.603525 11343 raft_consensus.cc:2781] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:19.603206 11352 consensus_queue.cc:260] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:19.604251 11343 raft_consensus.cc:686] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:19.605327 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.605295 11343 consensus_queue.cc:237] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.607326 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.604317 11352 raft_consensus.cc:3037] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.609311 9934 raft_consensus.cc:1226] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:19.609650 9979 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:19.612300 11354 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.611512 9934 raft_consensus.cc:1226] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:19.614249 11354 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:19.615149 10082 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:19.617640 9979 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:19.619060 11236 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:19.584204 11358 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:19.619593 11230 raft_consensus.cc:2726] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 09:55:19.631935 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 43752us
I20240426 09:55:19.604561 11253 raft_consensus.cc:2781] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:19.643724 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.644461 10014 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.646248 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.647500 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.648000 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.648391 10015 raft_consensus.cc:1261] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240426 09:55:19.649303 9979 leader_election.cc:334] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 1.842s (SENT)
I20240426 09:55:19.652909 10007 raft_consensus.cc:1226] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:19.649273 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.653951 10010 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:19.631527 11236 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:19.659010 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.660647 9906 leader_election.cc:398] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:19.661953 9906 leader_election.cc:401] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 09:55:19.662595 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:55:19.661798 9982 leader_election.cc:334] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Timed out: RequestConsensusVote RPC to 127.8.141.129:38113 timed out after 2.364s (SENT)
I20240426 09:55:19.665625 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.666656 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.667485 9937 raft_consensus.cc:2379] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.667296 11328 consensus_queue.cc:1046] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:19.668078 9906 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:19.668341 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.668622 11320 raft_consensus.cc:3032] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:19.656771 9934 raft_consensus.cc:1226] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:19.669062 9937 raft_consensus.cc:2454] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:19.670078 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.670910 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.671970 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.672502 9933 raft_consensus.cc:2454] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:19.636864 11253 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:19.673010 11253 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:19.673261 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.674317 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.666960 11350 raft_consensus.cc:3037] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.676726 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.678148 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.678866 9940 raft_consensus.cc:2379] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.681248 9933 raft_consensus.cc:2379] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:19.669992 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.681951 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.665194 11236 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.688323 9934 raft_consensus.cc:2379] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:19.690558 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.691427 9934 raft_consensus.cc:2379] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.692224 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.693378 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.694021 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.694674 9936 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:19.695591 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.696285 9936 raft_consensus.cc:2379] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.669027 11320 raft_consensus.cc:729] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:19.697010 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.697106 11320 consensus_queue.cc:260] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.698133 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.698354 11320 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:19.681792 11331 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:19.699587 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.702200 9932 raft_consensus.cc:2379] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.703002 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.703262 9979 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:19.699899 9935 raft_consensus.cc:1226] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:19.705904 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.700310 11331 raft_consensus.cc:686] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:19.707512 11331 consensus_queue.cc:237] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.668397 11328 consensus_queue.cc:1035] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20240426 09:55:19.700037 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.711854 9938 raft_consensus.cc:2379] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.712548 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.700987 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.716789 9940 raft_consensus.cc:2379] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.717597 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.718623 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.725013 9940 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.725903 10012 raft_consensus.cc:3032] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:19.698747 9936 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:19.737725 10620 raft_consensus.cc:1226] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEADER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:19.741312 9938 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:19.743228 9979 leader_election.cc:302] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:19.744143 11230 raft_consensus.cc:2726] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:19.726898 10012 raft_consensus.cc:729] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:19.746855 10012 consensus_queue.cc:260] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:19.747905 10012 raft_consensus.cc:3037] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.751178 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.752403 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.757295 9935 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:19.742604 10014 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:19.742373 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.725689 9937 raft_consensus.cc:2354] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
I20240426 09:55:19.701359 9934 raft_consensus.cc:2396] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240426 09:55:19.779827 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.781111 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.782286 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.783803 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.784636 9937 raft_consensus.cc:2379] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.761294 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.785912 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.787447 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.760923 11337 raft_consensus.cc:2741] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20240426 09:55:19.789166 9937 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:19.790048 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.763226 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.790789 10008 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:19.765350 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.794916 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.795558 10011 raft_consensus.cc:2354] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:19.796221 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.766091 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.802881 10007 raft_consensus.cc:2362] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2. Re-sending same reply.
I20240426 09:55:19.804348 10007 raft_consensus.cc:1261] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20240426 09:55:19.804673 10007 pending_rounds.cc:76] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting all ops after (but not including) 4
I20240426 09:55:19.804869 10007 pending_rounds.cc:97] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20240426 09:55:19.805106 10007 raft_consensus.cc:2866] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:19.767037 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.807140 10009 raft_consensus.cc:2379] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:19.807564 11352 consensus_queue.cc:1035] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:19.767696 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.766716 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:19.812793 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.813544 9933 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:19.788502 9934 raft_consensus.cc:2379] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:19.786562 9935 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
W20240426 09:55:19.790308 9904 service_pool.cc:135] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:47242 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 4 similar messages]
I20240426 09:55:19.776479 9932 raft_consensus.cc:2454] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:19.759650 9940 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:19.762229 10010 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:19.810485 9907 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:40928 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1136)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 963)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1129)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 964)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1132)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1142)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1143)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 966)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1144)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 967)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1145)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 968)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1146)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 969)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 970)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 971)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1130)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 972)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1134)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 973)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1133)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1147)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1148)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1140)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 974)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1131)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 975)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1138)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 976)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 977)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 978)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 979)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 980)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 981)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 965)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 926)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 927)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 929)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 930)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 935)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 105)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 949)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 950)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 951)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 952)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 953)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 959)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 106)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 107)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 108)
W20240426 09:55:19.822585 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 167830us
I20240426 09:55:19.821800 9906 leader_election.cc:398] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:19.824791 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.825721 9936 raft_consensus.cc:2396] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 09:55:19.824086 10007 raft_consensus.cc:1261] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:19.824050 10012 raft_consensus.cc:1261] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240426 09:55:19.825654 9932 raft_consensus.cc:2379] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.828016 11352 consensus_queue.cc:1035] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240426 09:55:19.830801 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 120627us
I20240426 09:55:19.829113 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.833130 10011 raft_consensus.cc:1226] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 09:55:19.832269 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.832348 9906 leader_election.cc:401] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 09:55:19.838927 11331 raft_consensus.cc:2726] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:19.831596 9904 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:47242 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1132)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1142)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1143)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 966)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1144)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 967)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1145)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 968)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1146)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 969)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 970)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 971)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1130)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 972)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1134)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 973)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1133)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1147)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1148)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1140)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 974)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1131)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 975)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1138)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 976)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 977)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 978)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 979)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 980)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 981)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 965)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 926)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 927)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 929)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 930)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 935)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 105)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 949)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 950)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 951)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 952)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 953)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 959)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 106)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 107)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 108)
I20240426 09:55:19.846392 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.841389 9906 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:19.849993 10056 leader_election.cc:334] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Timed out: RequestConsensusVote RPC to 127.8.141.129:38113 timed out after 1.621s (SENT)
I20240426 09:55:19.852162 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.852825 9933 raft_consensus.cc:2379] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.833997 9937 raft_consensus.cc:2362] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3. Re-sending same reply.
I20240426 09:55:19.854135 10090 raft_consensus.cc:1226] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:19.859946 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.860579 9937 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:19.861411 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.865900 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.865082 10081 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:19.867029 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.853713 9932 raft_consensus.cc:2379] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.867720 9941 raft_consensus.cc:2379] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.868070 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.868592 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.869165 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.869278 11236 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.869663 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.869966 9932 raft_consensus.cc:2379] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:19.870404 10013 raft_consensus.cc:2379] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:19.851146 10056 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:19.854833 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.870750 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.871289 11315 raft_consensus.cc:2726] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:19.873603 9932 raft_consensus.cc:2379] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.874234 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.874337 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.876021 9932 raft_consensus.cc:2379] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.876856 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.874750 11236 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:19.874743 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.868602 10008 raft_consensus.cc:2454] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:19.862432 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.878849 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.879094 9937 raft_consensus.cc:2379] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.879458 9933 raft_consensus.cc:2454] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:19.879926 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.880328 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.880676 9937 raft_consensus.cc:2379] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.854411 11331 raft_consensus.cc:2781] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:19.894588 11331 raft_consensus.cc:686] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:19.895455 11331 consensus_queue.cc:237] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.896821 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:19.872805 10056 leader_election.cc:334] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Timed out: RequestConsensusVote RPC to 127.8.141.129:38113 timed out after 1.784s (SENT)
I20240426 09:55:19.849838 11253 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.869671 11350 raft_consensus.cc:2726] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:19.890594 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.891022 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:55:19.891713 10111 debug-util.cc:398] Leaking SignalData structure 0x7b08001ca300 after lost signal to thread 8758
I20240426 09:55:19.892060 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.893869 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.898281 10056 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:19.882566 9935 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:19.900079 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.905320 9940 raft_consensus.cc:2379] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.906213 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.902318 9906 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:19.906987 9936 raft_consensus.cc:2454] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:19.902284 9933 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:19.904616 9941 raft_consensus.cc:2362] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3. Re-sending same reply.
I20240426 09:55:19.906515 9938 raft_consensus.cc:2454] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:19.908859 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.909423 9936 raft_consensus.cc:2396] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240426 09:55:19.910573 9939 raft_consensus.cc:2379] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
W20240426 09:55:19.903434 10111 debug-util.cc:398] Leaking SignalData structure 0x7b0800182660 after lost signal to thread 8761
I20240426 09:55:19.877955 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.906275 11328 consensus_queue.cc:1035] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20240426 09:55:19.912918 10082 raft_consensus.cc:1226] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:19.913118 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.938598 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.914058 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.915997 10100 tablet_copy_service.cc:342] P d3a480e831d649139ff016fb68aee3ee: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-6151686b1a24447880e49b7d7395fcc1 received from {username='slave'} at 127.0.0.1:48202
I20240426 09:55:19.918162 10083 raft_consensus.cc:1226] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:19.919030 11375 raft_consensus.cc:2726] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:19.919477 10087 raft_consensus.cc:1226] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:19.940686 9939 raft_consensus.cc:2379] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.941804 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.941954 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.919591 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:19.920440 11253 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:19.911242 11350 raft_consensus.cc:2781] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:19.943401 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.921893 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.944314 10100 tablet_copy_service.cc:433] P d3a480e831d649139ff016fb68aee3ee: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-6151686b1a24447880e49b7d7395fcc1 on tablet 6151686b1a24447880e49b7d7395fcc1 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:19.944460 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:19.945120 10011 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:19.945308 10083 raft_consensus.cc:1261] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:19.945694 9938 raft_consensus.cc:2379] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.940769 11320 consensus_queue.cc:1046] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20240426 09:55:19.946501 10083 raft_consensus.cc:1226] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:19.946921 11320 raft_consensus.cc:3032] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:19.947304 11320 raft_consensus.cc:729] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:19.947851 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.948443 9932 raft_consensus.cc:2362] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3. Re-sending same reply.
I20240426 09:55:19.948032 11320 consensus_queue.cc:260] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.946513 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.927016 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:55:19.927340 10056 leader_election.cc:334] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Timed out: RequestConsensusVote RPC to 127.8.141.129:38113 timed out after 1.785s (SENT)
I20240426 09:55:19.930987 9934 raft_consensus.cc:2379] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.950721 10056 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:19.951424 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.952097 9938 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:19.952811 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.951323 11350 raft_consensus.cc:686] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:19.931416 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.931928 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:19.953699 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.954113 10086 raft_consensus.cc:2362] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3. Re-sending same reply.
I20240426 09:55:19.953872 9935 raft_consensus.cc:2362] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3. Re-sending same reply.
I20240426 09:55:19.932272 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.912161 10081 raft_consensus.cc:1226] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:19.933467 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:19.956656 10088 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:19.904942 9937 raft_consensus.cc:2379] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.938905 9933 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:19.940500 9941 raft_consensus.cc:2379] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.920917 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
W20240426 09:55:19.954456 10111 debug-util.cc:398] Leaking SignalData structure 0x7b08001d2f00 after lost signal to thread 9873
I20240426 09:55:19.958742 11350 consensus_queue.cc:237] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:19.952106 11375 raft_consensus.cc:2726] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:19.961160 10089 raft_consensus.cc:1226] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:19.951346 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.959097 10084 raft_consensus.cc:2379] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
W20240426 09:55:19.960325 10111 debug-util.cc:398] Leaking SignalData structure 0x7b08002ee020 after lost signal to thread 9965
I20240426 09:55:19.957624 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.960449 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.967732 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.969760 9935 raft_consensus.cc:2362] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3. Re-sending same reply.
I20240426 09:55:19.971367 10081 raft_consensus.cc:1226] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:19.971916 9939 raft_consensus.cc:2396] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 09:55:19.972700 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.973234 9939 raft_consensus.cc:2362] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3. Re-sending same reply.
I20240426 09:55:19.973945 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.974462 9939 raft_consensus.cc:2362] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3. Re-sending same reply.
I20240426 09:55:19.968575 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:19.978559 9938 raft_consensus.cc:2379] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:19.950006 11320 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:19.970649 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.979390 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.949189 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.980412 9940 raft_consensus.cc:2379] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:19.981990 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:19.983387 9941 raft_consensus.cc:2362] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3. Re-sending same reply.
I20240426 09:55:19.963301 10090 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
W20240426 09:55:19.994360 10111 debug-util.cc:398] Leaking SignalData structure 0x7b0800220bc0 after lost signal to thread 10040
W20240426 09:55:19.999271 10111 debug-util.cc:398] Leaking SignalData structure 0x7b08002e4900 after lost signal to thread 10114
I20240426 09:55:19.999563 9939 raft_consensus.cc:1226] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240426 09:55:20.001953 9939 raft_consensus.cc:1261] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 09:55:20.003434 9939 raft_consensus.cc:1261] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240426 09:55:20.006024 11352 consensus_queue.cc:1035] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:19.971606 9934 raft_consensus.cc:2379] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:20.008679 9938 raft_consensus.cc:1226] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:20.009670 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:20.012609 10090 raft_consensus.cc:2379] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 4: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:20.009881 10011 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:20.013437 9906 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:20.011634 9933 raft_consensus.cc:2454] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:20.016330 9933 raft_consensus.cc:1226] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:20.018590 9940 raft_consensus.cc:1226] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20240426 09:55:20.019822 9941 raft_consensus.cc:1226] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
W20240426 09:55:19.999639 10111 debug-util.cc:398] Leaking SignalData structure 0x7b08002e43e0 after lost signal to thread 10115
I20240426 09:55:20.021108 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:20.023064 10056 leader_election.cc:334] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Timed out: RequestConsensusVote RPC to 127.8.141.129:38113 timed out after 1.836s (SENT)
I20240426 09:55:20.023407 10056 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.027709 9939 raft_consensus.cc:1226] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:20.027308 9933 raft_consensus.cc:1261] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 09:55:20.029075 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:20.029829 10090 raft_consensus.cc:2379] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:20.020843 11331 raft_consensus.cc:2781] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:20.025909 9932 raft_consensus.cc:2379] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:20.046416 9933 raft_consensus.cc:1226] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:20.032668 9941 raft_consensus.cc:1226] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:20.052376 9939 raft_consensus.cc:1226] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:20.052403 10090 raft_consensus.cc:1226] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:20.053102 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:20.056629 9935 raft_consensus.cc:1226] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:20.045176 9934 raft_consensus.cc:1226] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:20.059792 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:20.058164 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:20.061646 9941 raft_consensus.cc:2379] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:20.042742 11350 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:20.032682 11361 raft_consensus.cc:2726] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:20.080024 9938 raft_consensus.cc:1261] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20240426 09:55:20.081151 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 63885.2us
I20240426 09:55:20.081362 9938 raft_consensus.cc:1226] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
W20240426 09:55:20.058164 10111 debug-util.cc:398] Leaking SignalData structure 0x7b08002ac7e0 after lost signal to thread 10644
I20240426 09:55:20.083083 9935 raft_consensus.cc:1226] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:20.054517 11331 raft_consensus.cc:686] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:20.087445 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:20.088277 10089 raft_consensus.cc:2379] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:20.032275 11037 consensus_queue.cc:1046] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20240426 09:55:20.090484 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:20.082283 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:20.097973 11331 consensus_queue.cc:237] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:20.086356 10111 debug-util.cc:398] Leaking SignalData structure 0x7b08000fe320 after lost signal to thread 10645
I20240426 09:55:20.101507 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:20.104086 9941 raft_consensus.cc:2379] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:20.110268 10089 raft_consensus.cc:1261] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:20.115615 10087 raft_consensus.cc:1261] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:20.108301 11381 consensus_queue.cc:1046] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20240426 09:55:20.117285 10088 raft_consensus.cc:1226] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:20.117699 11381 raft_consensus.cc:3032] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:20.118072 11381 raft_consensus.cc:729] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:20.118629 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:20.119413 10084 raft_consensus.cc:2354] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:20.118852 11381 consensus_queue.cc:260] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.124684 11381 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:20.124994 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:20.125218 9982 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.121235 10056 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.123646 11337 consensus_queue.cc:1046] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240426 09:55:20.112084 10111 debug-util.cc:398] Leaking SignalData structure 0x7b0800307d60 after lost signal to thread 11079
I20240426 09:55:20.127888 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:20.082505 11350 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.136266 11350 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:20.130400 10089 raft_consensus.cc:2379] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 4: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:20.134085 10056 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.137125 9904 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.134479 11336 raft_consensus.cc:2726] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:20.137651 10056 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.115301 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:20.115509 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:20.145924 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:20.146706 10011 raft_consensus.cc:2396] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 09:55:20.144423 11230 consensus_queue.cc:1046] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240426 09:55:20.146932 11253 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:20.147405 11253 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.142974 11236 raft_consensus.cc:2781] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:20.149144 11236 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:20.149469 11236 raft_consensus.cc:3037] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:20.151322 11277 raft_consensus.cc:3032] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:20.151693 11277 raft_consensus.cc:729] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:20.148857 11253 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:20.151789 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:20.159278 10010 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:20.165329 9904 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:20.148907 9934 raft_consensus.cc:2379] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:20.145814 11375 raft_consensus.cc:2726] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:20.196317 9838 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50254 (request call id 189) took 1122 ms. Trace:
I20240426 09:55:20.166494 10014 raft_consensus.cc:1226] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:20.167032 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:20.197708 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:20.167912 10056 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:20.169114 10007 raft_consensus.cc:1226] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:20.230638 10088 raft_consensus.cc:2379] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:20.145503 11350 raft_consensus.cc:2726] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:20.193883 9904 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:20.235847 10011 raft_consensus.cc:2379] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:20.196718 10009 raft_consensus.cc:1261] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:20.140461 11356 consensus_queue.cc:1046] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.099s
I20240426 09:55:20.194451 11253 raft_consensus.cc:2781] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:20.145272 11395 consensus_queue.cc:1035] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
I20240426 09:55:20.193915 11343 raft_consensus.cc:2726] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:20.230495 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:20.248495 9839 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58806 (request call id 84) took 1084 ms. Trace:
I20240426 09:55:20.253000 9839 rpcz_store.cc:276] 0426 09:55:19.163704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:19.163854 (+ 150us) service_pool.cc:225] Handling call
0426 09:55:20.248429 (+1084575us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46840,"reported_tablets":9}
W20240426 09:55:20.255995 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 88118.4us
I20240426 09:55:20.257073 10056 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.259076 11315 raft_consensus.cc:2726] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:20.197324 9838 rpcz_store.cc:276] 0426 09:55:19.073753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:19.073970 (+ 217us) service_pool.cc:225] Handling call
0426 09:55:20.196245 (+1122275us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:19.584705 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:19.585256 (+ 551us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:19.585311 (+ 55us) write_op.cc:427] Acquired schema lock
0426 09:55:19.585328 (+ 17us) tablet.cc:660] Decoding operations
0426 09:55:19.586384 (+ 1056us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:19.586490 (+ 106us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:19.586517 (+ 27us) tablet.cc:683] Acquiring locks for 12 operations
0426 09:55:19.695046 (+108529us) tablet.cc:699] Row locks acquired
0426 09:55:19.695088 (+ 42us) write_op.cc:260] PREPARE: finished
0426 09:55:19.695248 (+ 160us) write_op.cc:270] Start()
0426 09:55:19.695396 (+ 148us) write_op.cc:276] Timestamp: P: 1714125319695190 usec, L: 0
0426 09:55:19.695418 (+ 22us) op_driver.cc:331] REPLICATION: starting
0426 09:55:19.696047 (+ 629us) log.cc:842] Serialized 4858 byte log entry
0426 09:55:19.933132 (+237085us) op_driver.cc:444] REPLICATION: finished
0426 09:55:20.045064 (+111932us) write_op.cc:301] APPLY: starting
0426 09:55:20.045143 (+ 79us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:20.045369 (+ 226us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:20.045509 (+ 140us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:20.088122 (+ 42613us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:20.088344 (+ 222us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
0426 09:55:20.088382 (+ 38us) write_op.cc:312] APPLY: finished
0426 09:55:20.096440 (+ 8058us) log.cc:842] Serialized 88 byte log entry
0426 09:55:20.096593 (+ 153us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:20.096723 (+ 130us) write_op.cc:446] Released schema lock
0426 09:55:20.096770 (+ 47us) write_op.cc:341] FINISH: Updating metrics
0426 09:55:20.097023 (+ 253us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:20.097054 (+ 31us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":8333,"reported_tablets":21,"child_traces":[["op",{"apply.queue_time_us":111674,"apply.run_cpu_time_us":3866,"apply.run_wall_time_us":52382,"num_ops":12,"prepare.queue_time_us":163921,"prepare.run_cpu_time_us":32914,"prepare.run_wall_time_us":239211,"raft.queue_time_us":105048,"raft.run_cpu_time_us":9356,"raft.run_wall_time_us":91438,"replication_time_us":237629,"thread_start_us":433076,"threads_started":4,"wal-append.queue_time_us":129310}]]}
I20240426 09:55:20.197405 11132 consensus_queue.cc:1046] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20240426 09:55:20.152509 11277 consensus_queue.cc:260] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:20.251677 11328 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:20.255909 11375 raft_consensus.cc:2781] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:20.250332 11253 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:20.264837 9936 raft_consensus.cc:2396] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20240426 09:55:20.282140 11400 log.cc:924] Time spent T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.086s user 0.000s sys 0.001s
I20240426 09:55:20.281898 9836 catalog_manager.cc:5484] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
W20240426 09:55:20.289474 11393 log.cc:924] Time spent T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20240426 09:55:20.262323 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 107067us
I20240426 09:55:20.305114 9836 catalog_manager.cc:5484] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:20.294692 11236 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.324744 11381 raft_consensus.cc:3032] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:20.331836 11381 raft_consensus.cc:729] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
W20240426 09:55:20.332468 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 101690us
I20240426 09:55:20.321009 11277 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:20.305908 11253 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:20.301157 11375 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:20.321787 9836 catalog_manager.cc:5484] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:20.332736 11381 consensus_queue.cc:260] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.341106 11375 raft_consensus.cc:3037] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:20.341308 11381 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:20.300613 11328 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:20.334151 11236 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:20.343338 11328 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
W20240426 09:55:20.362545 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 105538us
I20240426 09:55:20.363407 10056 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.371089 9935 raft_consensus.cc:1226] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
W20240426 09:55:20.376729 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 182911us
W20240426 09:55:20.378063 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 224148us
I20240426 09:55:20.371713 11343 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:20.371366 11352 consensus_queue.cc:1046] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20240426 09:55:20.387195 10583 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:20.388844 11337 raft_consensus.cc:3032] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:20.389221 11337 raft_consensus.cc:729] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:20.390118 11337 consensus_queue.cc:260] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.376245 11405 consensus_queue.cc:1035] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20240426 09:55:20.391713 11337 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:20.385339 9904 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.395871 9979 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:20.387576 11230 raft_consensus.cc:3032] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:20.401904 11230 raft_consensus.cc:729] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:20.402706 11230 consensus_queue.cc:260] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.403614 10055 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:20.403816 11230 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:20.405314 11343 raft_consensus.cc:2726] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:20.407153 11403 raft_consensus.cc:2781] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader election won for term 4
W20240426 09:55:20.305466 11395 consensus_queue.cc:1159] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d3a480e831d649139ff016fb68aee3ee" responder_term: 3 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 09:55:20.385097 11394 consensus_queue.cc:1046] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20240426 09:55:20.409283 10055 leader_election.cc:302] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:20.410421 9979 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:20.411803 10055 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:20.400941 11320 raft_consensus.cc:2726] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:20.415138 11410 raft_consensus.cc:2781] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:20.415457 10081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:20.416146 10081 raft_consensus.cc:595] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.414875 11315 consensus_queue.cc:1035] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240426 09:55:20.413355 11343 raft_consensus.cc:2726] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:20.442724 10055 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.448480 9904 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.439548 11336 raft_consensus.cc:2726] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:20.417877 11375 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.451087 10055 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
W20240426 09:55:20.451218 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 73673us
W20240426 09:55:20.451972 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 326830us
I20240426 09:55:20.452244 10056 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.454052 9979 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.440963 11410 raft_consensus.cc:686] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:20.455904 9904 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.456913 10055 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:20.458909 11361 raft_consensus.cc:2726] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:20.459439 11336 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:20.459959 11336 raft_consensus.cc:686] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:20.460795 11336 consensus_queue.cc:237] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.462917 11404 raft_consensus.cc:2726] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:20.463963 10055 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:20.464660 11404 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:20.465086 11404 raft_consensus.cc:686] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:20.465844 11404 consensus_queue.cc:237] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.469810 11343 raft_consensus.cc:2726] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:20.471837 9979 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.476763 9904 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.485234 9904 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.488436 11408 raft_consensus.cc:2781] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:20.483799 11361 consensus_queue.cc:1046] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20240426 09:55:20.488281 11331 raft_consensus.cc:2726] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:20.489380 11361 raft_consensus.cc:3032] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:20.489754 11361 raft_consensus.cc:729] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:20.489851 11410 consensus_queue.cc:237] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.491289 11361 consensus_queue.cc:260] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.492398 11361 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:20.455369 11320 raft_consensus.cc:2726] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:20.444923 11315 consensus_queue.cc:1046] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20240426 09:55:20.497530 9979 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.474035 11236 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:20.453868 11356 consensus_queue.cc:1046] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20240426 09:55:20.484061 11337 raft_consensus.cc:2726] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240426 09:55:20.504397 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 52213.4us
I20240426 09:55:20.504653 11343 consensus_queue.cc:1035] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:20.505896 11407 consensus_queue.cc:1035] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
W20240426 09:55:20.507130 11413 log.cc:924] Time spent T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.059s user 0.000s sys 0.001s
I20240426 09:55:20.504261 11406 raft_consensus.cc:2726] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:20.494416 11408 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:20.496291 11037 raft_consensus.cc:3032] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:20.482880 11403 raft_consensus.cc:686] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 4 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:20.504516 11411 consensus_queue.cc:1046] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20240426 09:55:20.498955 11350 raft_consensus.cc:2726] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:20.534651 10056 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:20.554839 11236 raft_consensus.cc:2726] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:20.535604 11337 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:55:20.560357 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 75202.6us
W20240426 09:55:20.562152 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 64584.4us
I20240426 09:55:20.563661 9904 leader_election.cc:398] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:20.543439 11408 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
W20240426 09:55:20.565896 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 52650.4us
I20240426 09:55:20.566725 9906 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:55:20.568135 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 115501us
I20240426 09:55:20.543711 11037 raft_consensus.cc:729] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:20.564437 9904 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.571508 10015 raft_consensus.cc:1261] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:20.576014 9904 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.583837 10009 raft_consensus.cc:1261] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:20.586405 11402 raft_consensus.cc:3032] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:20.577426 11331 raft_consensus.cc:2726] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:20.590168 11423 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:20.590641 11423 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.593516 11423 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:20.595803 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 83030.3us
I20240426 09:55:20.596561 10056 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.568872 11350 raft_consensus.cc:2781] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:20.612035 9904 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.578734 11411 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:20.612995 9982 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.581166 11320 raft_consensus.cc:2726] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:20.618124 9904 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.619727 9904 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:20.598543 11402 raft_consensus.cc:729] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:20.608398 11350 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:20.612372 11375 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:20.614540 11411 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:20.591346 11422 consensus_queue.cc:1046] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20240426 09:55:20.623520 11331 raft_consensus.cc:2726] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:20.627238 11320 raft_consensus.cc:2781] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:20.626865 11350 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:20.626600 11407 raft_consensus.cc:2726] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:20.622828 11403 consensus_queue.cc:237] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:20.643453 10056 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:20.643620 10015 raft_consensus.cc:1261] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:20.648048 10081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:20.648617 11424 raft_consensus.cc:2726] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:20.648713 11236 raft_consensus.cc:3032] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:20.649405 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:20.631927 11411 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:20.660696 10012 raft_consensus.cc:1226] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:20.672281 11419 raft_consensus.cc:3032] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:20.672628 11336 raft_consensus.cc:3032] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:20.673280 10081 raft_consensus.cc:595] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:20.687306 10089 raft_consensus.cc:1226] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:20.676128 11425 raft_consensus.cc:2726] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:20.675700 11236 raft_consensus.cc:729] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:20.738575 11404 raft_consensus.cc:2932] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:20.739768 11236 consensus_queue.cc:260] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.740922 11236 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:20.695674 11428 consensus_queue.cc:1035] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20240426 09:55:20.698448 10056 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
W20240426 09:55:20.743402 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 45008.6us
I20240426 09:55:20.743623 11428 raft_consensus.cc:2726] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:20.744738 10056 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.698726 11429 raft_consensus.cc:2781] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:20.620815 11037 consensus_queue.cc:260] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:20.745473 11253 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.676708 11402 consensus_queue.cc:260] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:20.746192 11428 raft_consensus.cc:2726] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:20.704849 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:20.706921 11356 raft_consensus.cc:3032] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:20.708405 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:20.747042 11356 raft_consensus.cc:729] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:20.747246 10084 raft_consensus.cc:2454] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:20.652122 11320 raft_consensus.cc:686] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:20.748363 9979 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:20.748551 11320 consensus_queue.cc:237] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.749248 11432 raft_consensus.cc:2781] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:20.749640 11432 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:20.749922 11432 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:20.637818 11406 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:20.728557 11419 raft_consensus.cc:729] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:20.751794 11406 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.752346 11419 consensus_queue.cc:260] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.753436 11419 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:20.731804 11336 raft_consensus.cc:729] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:20.754787 11406 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:20.727159 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 107537us
I20240426 09:55:20.730929 11435 consensus_queue.cc:1035] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 09:55:20.659850 11343 consensus_queue.cc:1035] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20240426 09:55:20.755581 11253 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:20.704424 11416 raft_consensus.cc:3032] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:20.756592 11416 raft_consensus.cc:729] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:20.757411 11416 consensus_queue.cc:260] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:20.758633 11416 raft_consensus.cc:3037] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:20.760571 11356 consensus_queue.cc:260] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.737884 11430 consensus_queue.cc:1046] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 09:55:20.745929 11429 raft_consensus.cc:686] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:20.766474 11429 consensus_queue.cc:237] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.745895 10056 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.767972 11430 raft_consensus.cc:2726] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:20.768788 11402 raft_consensus.cc:3037] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:20.766791 11405 consensus_queue.cc:1046] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240426 09:55:20.770488 10082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:20.770821 11405 raft_consensus.cc:3032] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:20.771225 11405 raft_consensus.cc:729] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:20.771955 11405 consensus_queue.cc:260] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.773075 11405 raft_consensus.cc:3037] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:20.786751 9904 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:20.753228 11337 raft_consensus.cc:2726] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:20.818073 9904 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:20.828752 10056 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.830889 9904 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
W20240426 09:55:20.790442 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 153039us
I20240426 09:55:20.835814 9982 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.782591 11336 consensus_queue.cc:260] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.826560 11037 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:55:20.852061 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 177691us
I20240426 09:55:20.828280 11331 raft_consensus.cc:2781] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:20.792363 11356 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:20.861546 9982 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.862814 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:20.838219 11253 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:20.862040 9904 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:20.863909 10007 raft_consensus.cc:1261] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:20.849485 11406 raft_consensus.cc:2781] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:20.794296 11315 consensus_queue.cc:1046] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20240426 09:55:20.859524 11337 raft_consensus.cc:2781] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:20.793704 11343 consensus_queue.cc:1046] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
I20240426 09:55:20.864775 9904 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.794100 11430 consensus_queue.cc:1046] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20240426 09:55:20.865187 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:20.782063 10009 raft_consensus.cc:2932] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:20.838507 11375 consensus_queue.cc:1046] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20240426 09:55:20.871732 9982 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.862571 11336 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:20.863708 11331 raft_consensus.cc:686] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:20.868973 11320 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:20.866324 11361 raft_consensus.cc:2726] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:20.873524 11230 raft_consensus.cc:2726] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:20.880089 11406 raft_consensus.cc:686] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:20.899500 11432 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.901703 11432 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:20.881835 11337 raft_consensus.cc:686] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:20.910214 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:20.913187 10089 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 4
I20240426 09:55:20.912520 11320 raft_consensus.cc:686] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:20.915392 10088 raft_consensus.cc:2932] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:20.926510 9982 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.928066 10013 raft_consensus.cc:1261] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240426 09:55:20.908679 11315 consensus_queue.cc:1046] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20240426 09:55:20.881121 11410 raft_consensus.cc:2932] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:20.932258 9982 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.933111 11431 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:20.928138 11408 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.937438 11408 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:20.939872 11361 raft_consensus.cc:3032] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:20.940240 11361 raft_consensus.cc:729] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:20.940992 11361 consensus_queue.cc:260] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.942272 11361 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:20.916792 11444 raft_consensus.cc:2726] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:20.944108 11408 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:20.946259 11430 raft_consensus.cc:3032] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:20.946640 11430 raft_consensus.cc:729] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:20.947445 11430 consensus_queue.cc:260] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:20.948730 11430 raft_consensus.cc:3037] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:20.951411 9982 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.952348 11337 consensus_queue.cc:237] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:20.955442 11422 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:20.960601 9982 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:20.928663 11381 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240426 09:55:20.957008 11328 consensus_meta.cc:287] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.587s user 0.009s sys 0.002s
I20240426 09:55:20.968255 11328 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:20.977051 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 112353us
W20240426 09:55:21.003073 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 100889us
I20240426 09:55:21.003774 9904 leader_election.cc:398] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3.
I20240426 09:55:21.004848 9904 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:21.007239 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 47943.7us
I20240426 09:55:21.006017 10089 raft_consensus.cc:2454] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:21.008365 9982 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:21.008888 9979 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:20.963851 11424 raft_consensus.cc:3032] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:21.008823 9904 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:20.919435 11331 consensus_queue.cc:237] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:20.969697 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:21.011703 10081 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:20.970130 11381 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:20.963163 11406 consensus_queue.cc:237] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.029959 9982 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:21.030308 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:21.000311 11236 raft_consensus.cc:2726] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:21.029124 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:21.035141 9938 raft_consensus.cc:2396] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 09:55:21.037788 11445 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:21.039580 11445 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.042649 11427 raft_consensus.cc:2781] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:21.034333 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:20.977728 11428 consensus_queue.cc:1035] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20240426 09:55:21.069134 11422 raft_consensus.cc:2726] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:21.041590 11445 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:21.011278 11424 raft_consensus.cc:729] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:21.027732 11425 raft_consensus.cc:2726] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:21.029389 11381 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:21.030460 11407 raft_consensus.cc:2726] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 09:55:21.075336 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 45451.6us
I20240426 09:55:21.082676 9982 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:21.010495 11435 consensus_queue.cc:1046] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20240426 09:55:21.085368 11426 raft_consensus.cc:2726] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:21.007722 11444 consensus_queue.cc:1035] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
W20240426 09:55:21.086522 11446 log.cc:924] Time spent T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20240426 09:55:21.087182 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:21.087733 11452 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:21.099007 11452 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:21.105232 11447 log.cc:924] Time spent T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.096s user 0.001s sys 0.001s
I20240426 09:55:21.104900 11411 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:21.107308 11451 log.cc:924] Time spent T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.104s user 0.001s sys 0.000s
I20240426 09:55:21.075874 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:21.074204 11431 raft_consensus.cc:2726] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:21.089685 11425 raft_consensus.cc:2726] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:21.096072 10011 raft_consensus.cc:2379] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:21.031359 11320 consensus_queue.cc:237] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.096099 11429 raft_consensus.cc:3032] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:21.083839 11403 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:21.097771 11444 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:21.093125 11253 raft_consensus.cc:686] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:21.096779 11432 raft_consensus.cc:2726] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:21.129603 9982 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
W20240426 09:55:21.123555 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 76343.9us
I20240426 09:55:21.089753 11427 raft_consensus.cc:686] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:21.116235 10008 raft_consensus.cc:1226] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
W20240426 09:55:21.127705 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 248392us
I20240426 09:55:21.108704 11328 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:21.122660 11435 raft_consensus.cc:3032] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:21.122862 10620 raft_consensus.cc:2932] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:21.137440 10089 raft_consensus.cc:1261] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:21.124629 11429 raft_consensus.cc:729] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:21.132418 9932 raft_consensus.cc:2379] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:21.157459 11452 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:21.132943 10082 raft_consensus.cc:595] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:21.167397 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:21.169879 10010 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:21.170851 10007 raft_consensus.cc:1226] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:21.159083 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:21.172623 10084 raft_consensus.cc:1226] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:21.140936 11435 raft_consensus.cc:729] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:21.188115 9940 raft_consensus.cc:1226] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:21.188459 11435 consensus_queue.cc:260] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.188939 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:21.144743 11350 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.189559 9940 raft_consensus.cc:1226] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: [1.3-1.3]
I20240426 09:55:21.188611 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:21.191848 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:21.191112 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:21.145138 11410 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:21.183504 11236 consensus_queue.cc:1035] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:21.194157 9936 raft_consensus.cc:2379] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:21.185225 11337 consensus_queue.cc:1046] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:21.124186 11424 consensus_queue.cc:260] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.187223 11253 consensus_queue.cc:237] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.184341 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:21.196321 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:21.195362 9906 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:21.161578 11425 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:21.211695 9935 raft_consensus.cc:2379] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
W20240426 09:55:21.211908 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 82380.7us
I20240426 09:55:21.215338 9982 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:21.214095 9933 raft_consensus.cc:1261] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:21.216810 10007 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:21.213330 11337 raft_consensus.cc:3032] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:21.131474 11444 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.191095 11350 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:21.128165 11403 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:21.225411 10011 raft_consensus.cc:1226] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:21.229238 10015 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:21.211421 11331 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:21.224704 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:21.223407 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:21.243343 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 48053.1us
I20240426 09:55:21.241016 11429 consensus_queue.cc:260] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.223632 11337 raft_consensus.cc:729] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:21.226076 10614 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:21.228451 11407 raft_consensus.cc:2781] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:21.268116 9982 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:21.255281 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:21.238962 11328 raft_consensus.cc:2781] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:21.276326 9940 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:21.278009 10007 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:21.276360 9838 catalog_manager.cc:5484] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 3, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 3 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:21.236833 11411 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:21.267256 11456 log.cc:924] Time spent T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
I20240426 09:55:21.286325 9906 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:21.287559 10014 raft_consensus.cc:1261] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240426 09:55:21.333771 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 47535.3us
I20240426 09:55:21.266520 11435 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
W20240426 09:55:21.336392 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 68349.7us
I20240426 09:55:21.330165 11402 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:21.322095 10008 raft_consensus.cc:1226] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:21.279399 11407 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:21.269989 11403 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:21.339556 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 183410us
I20240426 09:55:21.331511 11408 raft_consensus.cc:2726] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:21.279354 11328 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:21.251852 10081 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:21.343021 9982 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:21.223793 11425 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.329955 11431 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
W20240426 09:55:21.343101 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 89806.8us
W20240426 09:55:21.354369 10056 leader_election.cc:334] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Timed out: RequestConsensusVote RPC to 127.8.141.129:38113 timed out after 2.415s (SENT)
I20240426 09:55:21.356731 9982 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:21.356956 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:21.246564 11331 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:21.279035 11406 consensus_queue.cc:1046] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20240426 09:55:21.349926 11407 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:21.330725 10613 raft_consensus.cc:2932] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:21.372861 11474 raft_consensus.cc:2781] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:21.373064 11464 consensus_queue.cc:1046] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20240426 09:55:21.373534 11474 raft_consensus.cc:686] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:21.374907 11474 consensus_queue.cc:237] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.368855 10009 raft_consensus.cc:1226] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:21.349412 11402 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:21.368551 11455 log.cc:924] Time spent T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.221s user 0.000s sys 0.001s
I20240426 09:55:21.358932 11230 raft_consensus.cc:2726] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:21.373279 11356 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:21.225700 11427 consensus_queue.cc:237] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.357313 11328 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:21.343897 11452 consensus_queue.cc:1035] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20240426 09:55:21.386204 9941 raft_consensus.cc:1261] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:21.391285 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:21.397159 9979 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:21.399027 10089 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:21.436672 10089 raft_consensus.cc:595] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:21.399405 10008 raft_consensus.cc:1226] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:21.399649 10081 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 09:55:21.400314 11444 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:21.400771 11402 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:21.367008 11419 consensus_meta.cc:287] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.613s user 0.005s sys 0.006s
I20240426 09:55:21.412887 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:21.413569 11424 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:21.413956 11406 consensus_queue.cc:1046] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240426 09:55:21.422432 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 72454us
I20240426 09:55:21.347785 11461 consensus_queue.cc:1035] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.257s
I20240426 09:55:21.424374 11381 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:21.427700 11336 consensus_meta.cc:287] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.546s user 0.008s sys 0.000s
I20240426 09:55:21.428122 9836 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50240 (request call id 199) took 1366 ms. Trace:
I20240426 09:55:21.428738 9838 catalog_manager.cc:5484] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 3, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 3 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:21.351258 11337 consensus_queue.cc:260] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:21.431164 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 45159.2us
I20240426 09:55:21.432776 11473 raft_consensus.cc:3032] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:21.411178 11458 consensus_queue.cc:1046] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.136s
W20240426 09:55:21.440992 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 43906.4us
I20240426 09:55:21.440596 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:21.449237 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:21.467660 10009 raft_consensus.cc:2454] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:55:21.452589 10015 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 09:55:21.468328 11381 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:21.461834 10016 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:55:21.469591 10583 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:21.469956 11331 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:21.453831 9836 rpcz_store.cc:276] 0426 09:55:20.061958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:20.062122 (+ 164us) service_pool.cc:225] Handling call
0426 09:55:21.428060 (+1365938us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:20.507485 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:20.507972 (+ 487us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:20.508009 (+ 37us) write_op.cc:427] Acquired schema lock
0426 09:55:20.508025 (+ 16us) tablet.cc:660] Decoding operations
0426 09:55:20.508583 (+ 558us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:20.508649 (+ 66us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:20.508678 (+ 29us) tablet.cc:683] Acquiring locks for 3 operations
0426 09:55:20.508975 (+ 297us) tablet.cc:699] Row locks acquired
0426 09:55:20.508993 (+ 18us) write_op.cc:260] PREPARE: finished
0426 09:55:20.509128 (+ 135us) write_op.cc:270] Start()
0426 09:55:20.509227 (+ 99us) write_op.cc:276] Timestamp: P: 1714125320509089 usec, L: 0
0426 09:55:20.509247 (+ 20us) op_driver.cc:331] REPLICATION: starting
0426 09:55:20.509849 (+ 602us) log.cc:842] Serialized 1316 byte log entry
0426 09:55:20.730268 (+220419us) op_driver.cc:444] REPLICATION: finished
0426 09:55:21.056200 (+325932us) write_op.cc:301] APPLY: starting
0426 09:55:21.071034 (+ 14834us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:21.083845 (+ 12811us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:21.107469 (+ 23624us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:21.107868 (+ 399us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:21.108029 (+ 161us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0426 09:55:21.108070 (+ 41us) write_op.cc:312] APPLY: finished
0426 09:55:21.108267 (+ 197us) log.cc:842] Serialized 34 byte log entry
0426 09:55:21.108410 (+ 143us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:21.116136 (+ 7726us) write_op.cc:446] Released schema lock
0426 09:55:21.118143 (+ 2007us) write_op.cc:341] FINISH: Updating metrics
0426 09:55:21.126631 (+ 8488us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:21.128307 (+ 1676us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":17639,"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":282395,"apply.run_cpu_time_us":2312,"apply.run_wall_time_us":99549,"num_ops":3,"prepare.queue_time_us":129241,"prepare.run_cpu_time_us":2632,"prepare.run_wall_time_us":2638,"replication_time_us":220934,"thread_start_us":255403,"threads_started":2}]]}
I20240426 09:55:21.468863 10013 raft_consensus.cc:1226] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:21.471374 11406 raft_consensus.cc:3032] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:21.471765 11406 raft_consensus.cc:729] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:21.472525 11406 consensus_queue.cc:260] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.473592 11406 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:21.446308 11404 raft_consensus.cc:3032] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:21.474954 11404 raft_consensus.cc:729] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:21.460767 11473 raft_consensus.cc:729] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:21.475827 11404 consensus_queue.cc:260] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:21.476249 11473 consensus_queue.cc:260] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.476909 11459 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:21.477416 11459 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.479357 11459 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:21.465521 9934 raft_consensus.cc:2454] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 1.
I20240426 09:55:21.465168 11343 raft_consensus.cc:979] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:21.455008 10056 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
W20240426 09:55:21.464797 11471 log.cc:924] Time spent T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.201s user 0.001s sys 0.000s
I20240426 09:55:21.459741 9982 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:21.449909 11230 raft_consensus.cc:2781] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:21.440852 11356 raft_consensus.cc:686] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:21.504304 11356 consensus_queue.cc:237] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.489524 11444 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:55:21.507064 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 47408.8us
I20240426 09:55:21.507404 11444 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.509331 11444 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:21.480501 11431 raft_consensus.cc:2726] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:21.515692 10614 raft_consensus.cc:1261] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240426 09:55:21.510445 11459 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:21.516397 11459 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.518437 11452 consensus_queue.cc:1035] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:21.484011 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:21.403468 11411 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20240426 09:55:21.461303 9937 raft_consensus.cc:1226] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:21.487212 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:21.490415 10088 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:21.492587 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:21.494246 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:21.495250 11410 raft_consensus.cc:2726] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:21.497500 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:21.530440 10010 raft_consensus.cc:1226] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:21.501137 11328 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:21.477028 11404 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:21.480746 11361 consensus_queue.cc:1046] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20240426 09:55:21.461525 9838 catalog_manager.cc:5484] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 3, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 3 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:21.512789 11236 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:21.513127 11230 raft_consensus.cc:686] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:21.509832 11461 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20240426 09:55:21.477532 11473 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
W20240426 09:55:21.520670 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 65744.5us
I20240426 09:55:21.554316 10056 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:21.518743 11459 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:21.523236 10088 raft_consensus.cc:595] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:21.556953 10088 raft_consensus.cc:603] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 09:55:21.526633 10081 raft_consensus.cc:2454] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:55:21.529207 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:21.558912 10089 raft_consensus.cc:2379] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:21.530076 10012 raft_consensus.cc:2454] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:21.547819 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:21.548345 9838 catalog_manager.cc:5484] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 3 to 4, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 4 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:21.522019 11375 raft_consensus.cc:1067] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:21.551368 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:21.527081 11403 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:21.521281 11343 raft_consensus.cc:2726] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:21.551456 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:21.521764 9982 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:21.521093 11336 consensus_queue.cc:1046] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20240426 09:55:21.534411 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:21.560530 11132 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:21.601791 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:21.560854 10085 raft_consensus.cc:1261] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:21.603120 9838 catalog_manager.cc:5484] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 3, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 3 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
W20240426 09:55:21.718958 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 197267us
I20240426 09:55:21.720367 11426 raft_consensus.cc:2726] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:21.723284 11425 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:21.725338 10088 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:55:21.726891 11472 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:21.521697 10585 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:21.560209 9906 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:55:21.727684 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 167541us
I20240426 09:55:21.691646 10089 raft_consensus.cc:1226] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:21.600970 11331 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:21.734082 10085 raft_consensus.cc:1226] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:21.729915 9904 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:21.732918 11482 log.cc:924] Time spent T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.180s user 0.001s sys 0.000s
I20240426 09:55:21.725510 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:21.726661 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:21.734160 11394 raft_consensus.cc:2781] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240426 09:55:21.734580 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 223270us
I20240426 09:55:21.730806 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:21.700405 11423 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:21.708159 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 09:55:21.748651 10056 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:21.751751 10016 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:55:21.790099 10016 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:21.790452 10016 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:21.791234 11423 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.783133 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:21.707914 11431 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:21.709815 11411 consensus_queue.cc:1046] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20240426 09:55:21.729074 11427 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:21.703732 11459 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:21.603410 11464 consensus_queue.cc:1035] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.125s
I20240426 09:55:21.728951 11350 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:21.788028 10086 raft_consensus.cc:1226] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:21.746755 11425 raft_consensus.cc:2781] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240426 09:55:21.815346 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 66764.4us
I20240426 09:55:21.698632 11444 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:21.794626 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:21.708189 9888 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 86603.5us
I20240426 09:55:21.746999 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:21.718933 10082 raft_consensus.cc:1261] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:21.751768 11394 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 09:55:21.795430 11477 log.cc:924] Time spent T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20240426 09:55:21.801028 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:21.802273 10010 raft_consensus.cc:1226] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
W20240426 09:55:21.798367 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 48675us
W20240426 09:55:21.810648 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 80814us
I20240426 09:55:21.749645 11331 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.800689 11474 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:21.825618 10085 raft_consensus.cc:2379] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 2: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:21.827049 10056 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:21.830313 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:21.833526 10086 raft_consensus.cc:1261] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240426 09:55:21.837256 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:21.837810 10620 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:21.838162 10090 raft_consensus.cc:1261] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:21.838076 9904 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:21.839656 9904 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:21.814756 11425 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:21.802644 11361 consensus_queue.cc:1046] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20240426 09:55:21.831493 11394 raft_consensus.cc:3037] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:21.865448 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:55:21.866894 11418 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20240426 09:55:21.871199 9904 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:21.870929 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 43950.4us
I20240426 09:55:21.877616 10056 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:21.787192 11230 consensus_queue.cc:237] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.840924 11328 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:21.876652 10620 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:21.878754 10056 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:21.840072 11405 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:21.802722 11431 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.814524 11459 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.868726 11425 raft_consensus.cc:3037] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:21.813009 11427 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.823141 11350 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.851420 11472 raft_consensus.cc:2726] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:21.842334 11419 consensus_queue.cc:1046] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20240426 09:55:21.832556 11336 consensus_queue.cc:1046] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20240426 09:55:21.882027 10056 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:21.830092 11444 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:21.892089 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:21.878876 11458 raft_consensus.cc:2726] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:21.879729 11375 raft_consensus.cc:2726] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240426 09:55:21.889019 11435 consensus_meta.cc:287] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.542s user 0.008s sys 0.000s
I20240426 09:55:21.888144 11328 raft_consensus.cc:2726] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:21.885700 11343 raft_consensus.cc:2726] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:21.905489 9905 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:21.895812 11423 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:21.909791 11431 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:21.910996 11459 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:21.903694 10016 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:22.014639 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 132678us
W20240426 09:55:21.912557 11407 consensus_meta.cc:287] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.511s user 0.009s sys 0.000s
I20240426 09:55:21.888517 11405 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:22.016543 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 134688us
I20240426 09:55:22.027868 11489 raft_consensus.cc:2726] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:22.014030 11472 raft_consensus.cc:2726] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:22.030217 10016 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:55:22.037179 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 166053us
I20240426 09:55:22.039407 10056 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:22.039731 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:22.040815 10620 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Advancing to term 2
W20240426 09:55:22.043115 11479 log.cc:924] Time spent T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.304s user 0.001s sys 0.001s
I20240426 09:55:22.008994 11481 raft_consensus.cc:2726] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:22.045262 9904 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:22.016214 11490 raft_consensus.cc:2726] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 09:55:22.038268 11487 log.cc:924] Time spent T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.160s user 0.001s sys 0.000s
I20240426 09:55:22.030035 11328 raft_consensus.cc:2781] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:22.052882 9982 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:22.057225 10620 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:22.027860 11444 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:22.058959 11480 log.cc:924] Time spent T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.248s user 0.001s sys 0.000s
I20240426 09:55:22.016762 11411 consensus_queue.cc:1046] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20240426 09:55:22.038908 11343 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.029578 11407 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.103322 11496 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
W20240426 09:55:22.103955 11486 log.cc:924] Time spent T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.155s user 0.000s sys 0.001s
W20240426 09:55:22.104218 11483 log.cc:924] Time spent T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.155s user 0.001s sys 0.000s
I20240426 09:55:22.104653 11454 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:22.113034 11492 log.cc:924] Time spent T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
W20240426 09:55:22.118312 11485 log.cc:924] Time spent T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.272s user 0.001s sys 0.000s
W20240426 09:55:22.129070 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 250363us
W20240426 09:55:22.256587 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 211380us
I20240426 09:55:22.042281 11489 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.017011 11410 consensus_queue.cc:1035] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20240426 09:55:22.044303 11336 consensus_queue.cc:1035] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20240426 09:55:22.258421 9904 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:22.183862 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 131052us
I20240426 09:55:22.051554 11461 consensus_queue.cc:1035] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20240426 09:55:22.186167 9906 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:22.098620 11481 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.231807 11490 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.244263 9980 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:22.197556 11375 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.193132 11464 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:22.195562 11431 raft_consensus.cc:2726] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:22.279058 11464 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.281162 11464 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:22.263226 10007 raft_consensus.cc:1226] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
W20240426 09:55:22.199026 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 101883us
W20240426 09:55:22.266422 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 81404.4us
I20240426 09:55:22.283022 11037 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.105971 11331 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.283684 11405 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.283516 11037 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.286163 11037 leader_election.cc:288] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.288231 11427 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.195082 11328 raft_consensus.cc:686] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
W20240426 09:55:22.269629 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 208601us
I20240426 09:55:22.270083 11504 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:22.263010 11350 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.271452 11230 raft_consensus.cc:2781] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:22.291369 11331 raft_consensus.cc:2726] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:22.293354 9982 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
W20240426 09:55:22.274212 11473 consensus_meta.cc:287] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.719s user 0.003s sys 0.004s
I20240426 09:55:22.202193 11444 raft_consensus.cc:2726] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 09:55:22.290638 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 92651.1us
W20240426 09:55:22.297547 11236 consensus_meta.cc:287] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.748s user 0.007s sys 0.000s
W20240426 09:55:22.265579 11406 consensus_meta.cc:287] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.792s user 0.003s sys 0.008s
I20240426 09:55:22.303570 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:22.304499 10016 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:22.305954 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:22.306609 10081 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:22.306746 11416 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.308233 9982 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:22.309288 10055 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:22.230901 11343 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.318698 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:22.319341 10090 raft_consensus.cc:3032] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:22.319785 10090 raft_consensus.cc:729] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:22.298285 11394 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.318900 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:22.293561 11464 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.360670 11498 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.307948 11490 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.264981 11489 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.320338 10090 consensus_queue.cc:260] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.364718 11328 consensus_queue.cc:237] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.307494 11236 raft_consensus.cc:2726] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 09:55:22.367408 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 106081us
I20240426 09:55:22.307439 11230 raft_consensus.cc:686] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:22.317154 11356 raft_consensus.cc:2726] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:22.318799 11502 raft_consensus.cc:2726] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:22.376456 11427 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.291146 11375 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:22.376827 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 93558.2us
I20240426 09:55:22.399109 9906 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:22.380455 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 341096us
I20240426 09:55:22.380527 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:22.289423 11481 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.400059 10016 raft_consensus.cc:2454] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:22.400091 10056 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:22.255740 11454 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.402757 11454 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:22.314316 11350 consensus_queue.cc:1035] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20240426 09:55:22.324172 10087 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 1.
I20240426 09:55:22.320068 11416 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.405593 11416 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.407054 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:22.407793 11230 consensus_queue.cc:237] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:22.408821 11404 consensus_meta.cc:287] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.873s user 0.006s sys 0.001s
I20240426 09:55:22.409627 11407 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.412137 11481 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:22.412595 11356 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:22.413334 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:22.413069 11356 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.413970 11427 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.416289 11427 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:22.390712 11498 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.418005 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:22.418609 10614 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:22.419806 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:22.422379 9979 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:22.420387 10620 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:22.422904 9980 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:22.413936 10016 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:22.425781 11506 raft_consensus.cc:2781] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:22.427240 10055 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:22.389343 10008 raft_consensus.cc:1226] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:22.434739 11415 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.434909 11461 raft_consensus.cc:2726] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:22.381667 11464 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.436578 9980 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:22.415508 11356 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:55:22.442008 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 180596us
I20240426 09:55:22.436204 11432 raft_consensus.cc:2781] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:22.429409 11473 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:22.437297 11320 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:22.443186 11502 raft_consensus.cc:2781] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:22.441663 11407 raft_consensus.cc:2781] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:22.455260 9979 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:22.441437 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:22.427901 10585 leader_election.cc:302] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:22.463156 11503 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.463420 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:22.464051 10081 raft_consensus.cc:1261] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:22.465196 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:22.452309 11481 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:22.466742 11489 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.470290 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:22.465812 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:22.476227 10085 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:22.448590 11432 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:22.474876 10012 raft_consensus.cc:1261] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:22.451859 11343 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:22.477308 10016 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:22.457615 11502 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:22.468029 11410 raft_consensus.cc:2781] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:22.459432 11407 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:22.435354 11415 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.474293 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:22.485896 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 85877.2us
I20240426 09:55:22.476374 11481 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:22.501227 10056 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:22.478658 11416 raft_consensus.cc:2781] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:22.482985 11432 raft_consensus.cc:3037] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:22.502199 10012 raft_consensus.cc:1261] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:22.468657 11473 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.485895 11502 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:22.476948 11498 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:22.483479 11489 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:22.477380 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 57802.9us
I20240426 09:55:22.487694 11410 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:22.469337 11320 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.491108 11407 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:22.487344 11394 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.504624 11481 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:22.558377 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:22.559667 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:22.503729 11416 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:22.561512 9936 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:22.562402 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:22.563513 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:22.563982 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:22.456588 11510 consensus_queue.cc:1035] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
W20240426 09:55:22.565250 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 104870us
I20240426 09:55:22.567230 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:22.555011 11410 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:22.567992 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:22.554364 11472 raft_consensus.cc:2726] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:22.569681 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:22.569283 9936 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:22.572665 9940 raft_consensus.cc:1261] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:22.573307 10012 raft_consensus.cc:3037] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:55:22.565750 11425 consensus_meta.cc:287] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.672s user 0.012s sys 0.007s
I20240426 09:55:22.575278 10016 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
W20240426 09:55:22.565373 11501 log.cc:924] Time spent T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240426 09:55:22.565809 11416 raft_consensus.cc:3037] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:22.580965 9940 raft_consensus.cc:1261] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20240426 09:55:22.581755 10015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1235) took 1057 ms. Trace:
I20240426 09:55:22.483346 11503 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
W20240426 09:55:22.574045 11499 log.cc:924] Time spent T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20240426 09:55:22.582383 9940 pending_rounds.cc:76] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9: Aborting all ops after (but not including) 1
I20240426 09:55:22.568449 11375 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:22.589650 9936 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:22.575922 11490 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.579998 10012 raft_consensus.cc:2454] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
I20240426 09:55:22.582388 10015 rpcz_store.cc:276] 0426 09:55:21.527830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:21.534256 (+ 6426us) service_pool.cc:225] Handling call
0426 09:55:21.553293 (+ 19037us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:21.560281 (+ 6988us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:21.560310 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:21.700015 (+139705us) log.cc:842] Serialized 26 byte log entry
0426 09:55:22.200004 (+499989us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:22.200042 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:22.200104 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:22.581617 (+381513us) raft_consensus.cc:1671] finished
0426 09:55:22.581630 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:22.581715 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14708,"raft.queue_time_us":4221,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":45068,"thread_start_us":452676,"threads_started":1,"wal-append.queue_time_us":557645}
I20240426 09:55:22.592131 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:22.579739 11464 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:22.593463 10015 raft_consensus.cc:1261] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240426 09:55:22.572325 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:22.584106 9940 pending_rounds.cc:97] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240426 09:55:22.594432 9940 raft_consensus.cc:2866] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:22.594861 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:22.583699 11507 consensus_queue.cc:1035] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
W20240426 09:55:22.585203 11511 log.cc:924] Time spent T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20240426 09:55:22.581071 11513 consensus_queue.cc:1035] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.111s
I20240426 09:55:22.598053 9982 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:22.584096 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:22.611553 10016 raft_consensus.cc:2379] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:22.573161 10083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 844) took 1033 ms. Trace:
I20240426 09:55:22.557354 11489 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.614827 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:22.615051 9939 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:22.621605 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:22.622187 10016 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:22.616861 11356 raft_consensus.cc:2781] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:22.614141 11507 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.614430 9934 raft_consensus.cc:3037] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:22.582871 11425 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.625778 9936 raft_consensus.cc:2932] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:22.636863 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:22.638582 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:22.625185 11513 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.621448 11405 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20240426 09:55:22.646615 11489 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.621381 10585 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:22.650761 11473 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.628537 11356 raft_consensus.cc:686] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:22.652365 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:22.653621 11410 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.643946 11524 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:22.659462 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:22.660193 10015 raft_consensus.cc:2379] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 2: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:22.656587 11473 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.661870 11524 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.663529 11416 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.665410 11416 leader_election.cc:288] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.643687 11513 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.651624 11415 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.667287 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:22.668390 11524 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:22.628247 11507 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.676537 11356 consensus_queue.cc:237] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.677682 10613 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45564 (request call id 191) took 1484 ms. Trace:
I20240426 09:55:22.645077 11517 consensus_queue.cc:1035] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20240426 09:55:22.681699 10012 raft_consensus.cc:2379] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:22.643615 11521 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:22.669575 11394 raft_consensus.cc:2781] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:22.678050 11507 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.687894 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:22.704648 9940 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:22.705001 11424 raft_consensus.cc:3032] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:22.705377 11424 raft_consensus.cc:729] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:22.706090 11424 consensus_queue.cc:260] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.692734 11430 raft_consensus.cc:3032] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:22.711800 11430 raft_consensus.cc:729] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:22.712618 11430 consensus_queue.cc:260] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.713805 11430 raft_consensus.cc:3037] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:22.693145 11524 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.715574 11524 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.703598 11394 raft_consensus.cc:686] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 4 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:22.728046 11394 consensus_queue.cc:237] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.678066 10613 rpcz_store.cc:276] 0426 09:55:21.192851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:21.193660 (+ 809us) service_pool.cc:225] Handling call
0426 09:55:21.227225 (+ 33565us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:21.284803 (+ 57578us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 09:55:21.400809 (+116006us) log.cc:842] Serialized 14 byte log entry
0426 09:55:21.552801 (+151992us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:21.556254 (+ 3453us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:55:21.558867 (+ 2613us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:22.677429 (+1118562us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b5400730354. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:22.677485 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:22.677625 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6445,"raft.run_wall_time_us":45759,"spinlock_wait_cycles":44062848,"thread_start_us":134194,"threads_started":1,"wal-append.queue_time_us":151402}
I20240426 09:55:22.680375 11425 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.680721 11320 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.741181 11452 consensus_queue.cc:1035] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:22.710270 9838 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50254 (request call id 190) took 2004 ms. Trace:
I20240426 09:55:22.692131 11429 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:22.704336 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:22.715737 11521 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.703835 11513 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.616183 10083 rpcz_store.cc:276] 0426 09:55:21.535878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:21.547138 (+ 11260us) service_pool.cc:225] Handling call
0426 09:55:21.691561 (+144423us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:21.788366 (+ 96805us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:21.791065 (+ 2699us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:21.791943 (+ 878us) log.cc:842] Serialized 26 byte log entry
0426 09:55:22.254118 (+462175us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:22.360636 (+106518us) log.cc:842] Serialized 14 byte log entry
0426 09:55:22.509241 (+148605us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:22.556252 (+ 47011us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:22.556805 (+ 553us) raft_consensus.cc:1671] finished
0426 09:55:22.561091 (+ 4286us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:22.567309 (+ 6218us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16805,"prepare.queue_time_us":164072,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":110039,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":18120,"thread_start_us":609142,"threads_started":3,"wal-append.queue_time_us":492872}
I20240426 09:55:22.719015 11506 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:22.751078 11506 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:22.719444 11432 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.695384 11423 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:22.661985 11473 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.723256 11463 raft_consensus.cc:3032] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:22.723258 11503 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.723704 11435 consensus_queue.cc:1035] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:22.732473 9941 raft_consensus.cc:1226] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:22.736176 11461 consensus_queue.cc:1035] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:22.736670 9906 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:22.725498 11524 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.761296 11463 raft_consensus.cc:729] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:22.742646 9838 rpcz_store.cc:276] 0426 09:55:20.705523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:20.705918 (+ 395us) service_pool.cc:225] Handling call
0426 09:55:22.710223 (+2004305us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:21.899838 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:21.906439 (+ 6601us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:21.906469 (+ 30us) write_op.cc:427] Acquired schema lock
0426 09:55:21.906482 (+ 13us) tablet.cc:660] Decoding operations
0426 09:55:21.907082 (+ 600us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:21.907134 (+ 52us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:21.907160 (+ 26us) tablet.cc:683] Acquiring locks for 5 operations
0426 09:55:21.907634 (+ 474us) tablet.cc:699] Row locks acquired
0426 09:55:21.907651 (+ 17us) write_op.cc:260] PREPARE: finished
0426 09:55:21.907769 (+ 118us) write_op.cc:270] Start()
0426 09:55:21.907860 (+ 91us) write_op.cc:276] Timestamp: P: 1714125321907733 usec, L: 0
0426 09:55:21.907878 (+ 18us) op_driver.cc:331] REPLICATION: starting
0426 09:55:21.908470 (+ 592us) log.cc:842] Serialized 2093 byte log entry
0426 09:55:22.289129 (+380659us) op_driver.cc:444] REPLICATION: finished
0426 09:55:22.504229 (+215100us) write_op.cc:301] APPLY: starting
0426 09:55:22.507086 (+ 2857us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:22.566564 (+ 59478us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:22.575809 (+ 9245us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:22.589726 (+ 13917us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:22.589905 (+ 179us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0426 09:55:22.589949 (+ 44us) write_op.cc:312] APPLY: finished
0426 09:55:22.590692 (+ 743us) log.cc:842] Serialized 46 byte log entry
0426 09:55:22.590880 (+ 188us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:22.591001 (+ 121us) write_op.cc:446] Released schema lock
0426 09:55:22.591040 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0426 09:55:22.591290 (+ 250us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:22.591328 (+ 38us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":157936,"reported_tablets":13,"child_traces":[["op",{"apply.queue_time_us":192269,"apply.run_cpu_time_us":3162,"apply.run_wall_time_us":93032,"num_ops":5,"prepare.queue_time_us":150880,"prepare.run_cpu_time_us":3387,"prepare.run_wall_time_us":16411,"replication_time_us":358772,"thread_start_us":192716,"threads_started":2}]]}
I20240426 09:55:22.752558 10089 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 856) took 1019 ms. Trace:
I20240426 09:55:22.763971 10089 rpcz_store.cc:276] 0426 09:55:21.726206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:21.736898 (+ 10692us) service_pool.cc:225] Handling call
0426 09:55:22.704198 (+967300us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b58000b0498. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:55:22.745601 (+ 41403us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26305920}
W20240426 09:55:22.758299 11474 consensus_meta.cc:287] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.915s user 0.010s sys 0.000s
I20240426 09:55:22.676496 11407 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.762128 11463 consensus_queue.cc:260] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.769495 11463 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:22.766582 11513 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.771157 11524 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.772677 11524 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.775167 11473 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.775939 11524 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:22.780740 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 328396us
I20240426 09:55:22.783617 10089 raft_consensus.cc:1261] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:22.771565 11461 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:22.784143 11461 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:22.784475 11461 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:22.786443 9904 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:22.792690 9904 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:22.792704 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:22.793779 9904 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:22.794147 10089 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:22.795106 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:22.795652 10089 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:22.768303 11474 raft_consensus.cc:2726] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:22.800137 9904 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:22.802291 9904 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:22.767035 9934 raft_consensus.cc:2454] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:22.780737 11454 raft_consensus.cc:2781] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:22.824997 10086 raft_consensus.cc:1261] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:22.791170 11404 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.759860 11423 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.828210 11407 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.798220 11524 raft_consensus.cc:2726] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:22.785825 11513 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.822103 11425 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:22.820272 11498 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.836468 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:22.824553 11454 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:22.844650 11525 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.844771 11454 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:22.831135 11404 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.839612 11513 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:55:22.849462 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 348338us
I20240426 09:55:22.826056 10008 raft_consensus.cc:1226] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240426 09:55:22.840145 11527 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.843168 9933 raft_consensus.cc:2454] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:22.822489 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:22.828998 11474 raft_consensus.cc:2726] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:22.841120 10054 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:22.833953 11320 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.802696 11473 raft_consensus.cc:2781] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:22.881618 11473 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:22.881979 11473 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:22.801944 11435 raft_consensus.cc:2726] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:22.820103 11503 consensus_queue.cc:1035] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
W20240426 09:55:22.846254 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 1.15276e+06us
I20240426 09:55:22.800995 11432 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:22.834476 11410 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.852731 11236 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:22.861646 11449 raft_consensus.cc:2726] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:22.862380 9941 raft_consensus.cc:1226] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:22.862854 9940 raft_consensus.cc:1226] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
W20240426 09:55:22.866622 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 45702.9us
I20240426 09:55:22.862408 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:22.883612 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:22.850559 11423 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.884577 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:22.909098 10013 raft_consensus.cc:1226] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:22.909401 9979 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:22.833986 11481 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.855242 11426 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.887799 11521 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:22.884280 11361 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.897212 11429 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:22.896342 11462 raft_consensus.cc:2726] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:22.845584 11502 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.896734 11508 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.868183 10083 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:22.847820 11498 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.845135 11525 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.878942 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:22.929260 10012 raft_consensus.cc:3037] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 4
I20240426 09:55:22.870206 11527 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.903302 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:22.868093 11320 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.915746 10013 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1238) took 1077 ms. Trace:
I20240426 09:55:22.935138 10013 rpcz_store.cc:276] 0426 09:55:21.832484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:21.832596 (+ 112us) service_pool.cc:225] Handling call
0426 09:55:22.908777 (+1076181us) spinlock_profiling.cc:243] Waited 5.47 ms on lock 0x7b580004ec98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:55:22.908945 (+ 168us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:22.909889 (+ 944us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:22.910038 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12585216}
I20240426 09:55:22.894935 11445 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:22.917656 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:22.888170 11419 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.900012 11402 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:22.918928 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:22.909310 11503 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:22.927320 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 246132us
I20240426 09:55:22.931924 9932 raft_consensus.cc:2396] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 09:55:22.916879 11405 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:22.892160 11375 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:22.915805 10613 raft_consensus.cc:2454] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:22.941218 9941 raft_consensus.cc:2379] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:22.946053 9940 raft_consensus.cc:2454] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:22.915591 11430 consensus_queue.cc:1046] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20240426 09:55:22.875965 11520 consensus_queue.cc:1035] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20240426 09:55:22.955621 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:22.954082 9979 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:22.924372 11361 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.957010 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:22.958297 10013 raft_consensus.cc:2454] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:22.902626 11236 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.916746 11432 consensus_queue.cc:1035] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20240426 09:55:22.960757 9906 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:22.951781 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:22.962446 10086 raft_consensus.cc:2379] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:22.957757 11343 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:22.962999 11529 log.cc:924] Time spent T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240426 09:55:22.963346 11521 raft_consensus.cc:2781] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:22.955183 10009 raft_consensus.cc:2379] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 5: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:22.902791 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:22.959646 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:22.920992 11426 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.970201 9932 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
W20240426 09:55:22.965389 11532 log.cc:924] Time spent T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20240426 09:55:22.959441 11430 consensus_queue.cc:1225] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer f12521b6d51242629cf4297864df00b9 log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20240426 09:55:22.924358 11423 consensus_queue.cc:1035] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20240426 09:55:22.926144 11429 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.968751 10015 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:55:22.898396 11514 consensus_queue.cc:1046] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.159s
I20240426 09:55:22.966918 11431 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:22.966995 11521 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:22.968640 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:22.950493 11405 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:22.969048 9905 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:22.973657 11426 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:22.949450 11445 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.956981 11415 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:22.971359 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:22.981357 11425 raft_consensus.cc:2726] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:22.988031 10620 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:22.959573 11503 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:22.984524 9934 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:22.990788 11405 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:22.947914 11419 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.968537 11404 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:22.979588 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:22.957522 11375 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.993268 11435 raft_consensus.cc:2781] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:22.951503 11402 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:22.984377 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:22.968261 11343 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.015630 10012 raft_consensus.cc:2454] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
I20240426 09:55:22.991215 11431 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:23.033481 10056 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:23.036904 9980 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:22.991111 11521 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.037346 9940 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:22.939667 11508 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.037148 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:23.040737 10056 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:23.042353 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.043068 10056 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:23.044759 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.043018 10012 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:23.045533 10013 raft_consensus.cc:2454] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:23.013907 11502 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:23.046291 9906 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:23.046691 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.048134 10013 raft_consensus.cc:2396] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for term 3 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20240426 09:55:23.049620 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.050222 10620 raft_consensus.cc:1226] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:23.031508 11435 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:23.050498 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.013245 10083 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
I20240426 09:55:23.050100 10012 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:23.036486 11425 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:23.054625 9906 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:23.035987 11417 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:23.039983 9932 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.038039 11431 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.005260 11415 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:23.058367 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.047417 11253 raft_consensus.cc:2781] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:23.060493 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:22.984100 11462 consensus_queue.cc:1046] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20240426 09:55:23.044828 11481 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:23.049172 11430 raft_consensus.cc:2781] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:23.053179 11320 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.072271 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.064321 10583 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:23.073556 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:23.049118 11404 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:23.064024 10012 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:23.053697 11435 raft_consensus.cc:3037] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.053027 11498 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:23.047335 11518 raft_consensus.cc:2726] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:23.074594 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:23.072357 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:23.076234 10016 raft_consensus.cc:2379] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:23.060159 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:23.041488 10585 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:23.047662 11536 raft_consensus.cc:2781] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:23.077606 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:23.063217 11452 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:23.063045 11513 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:23.065557 11253 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:23.085604 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:23.163326 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.165906 10012 raft_consensus.cc:2396] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 09:55:23.076797 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:23.164149 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.074275 11356 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:23.068665 11361 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:23.164731 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 99403.4us
I20240426 09:55:23.166013 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:55:23.163879 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 77369.3us
I20240426 09:55:23.073228 11430 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:23.080468 11416 raft_consensus.cc:2781] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:23.060494 11425 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.180925 10016 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:23.075356 11404 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:23.079916 11452 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:23.187552 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.081764 11445 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:23.166857 11253 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.164551 11527 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:23.193908 11490 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:23.196137 11431 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.062633 11417 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:23.175459 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 136645us
I20240426 09:55:23.186216 11320 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:23.220392 9906 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:23.183831 10083 raft_consensus.cc:3037] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:23.197813 11430 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.165864 11236 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:23.183552 11474 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:23.222440 11404 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.183473 11394 raft_consensus.cc:2781] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:23.192381 11429 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:23.076206 11461 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.197033 11416 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:23.167004 11513 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:23.237900 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.266012 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.235994 11452 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:23.254631 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:23.246851 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:23.247200 11502 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:23.276527 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:23.192062 11356 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.278355 11356 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.278811 11356 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:23.257886 11525 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:23.279227 11356 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.266664 10016 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:23.281482 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.282032 10016 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:23.270170 10088 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:23.283494 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:23.284716 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:23.284560 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:23.277010 11502 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.285809 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:23.287223 11502 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.287655 10090 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:23.287781 10088 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:23.288573 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:23.289450 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:23.290350 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:23.256865 11474 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.223994 11506 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.248980 11343 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:23.250674 11375 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:55:23.272840 11538 log.cc:924] Time spent T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20240426 09:55:23.259092 11394 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:23.264880 11416 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:23.293901 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:23.261747 11419 leader_election.cc:288] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:23.260288 10585 leader_election.cc:302] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:23.272711 11417 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.303918 11425 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.343626 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:23.345146 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:23.345446 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.345949 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:23.304968 11431 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.347278 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:23.301291 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:23.347929 10618 raft_consensus.cc:2454] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:23.348598 9933 raft_consensus.cc:2379] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:23.349442 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:23.302240 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:23.349678 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:23.350564 10616 raft_consensus.cc:2379] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:23.350705 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:23.302618 9938 raft_consensus.cc:1261] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:23.290838 10088 raft_consensus.cc:3037] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:23.327911 10090 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:23.280347 11454 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.354247 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:23.249752 11320 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.263231 11503 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.337888 9936 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:23.306185 11461 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:23.350126 11463 consensus_meta.cc:287] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.580s user 0.010s sys 0.000s
I20240426 09:55:23.292382 11356 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.290746 11508 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:23.285348 11402 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.341574 10082 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:23.342320 11513 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:23.347823 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:23.246316 11490 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.294430 11428 raft_consensus.cc:3032] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:23.352135 9933 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:23.352615 10056 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:23.351416 10054 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:23.366889 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:23.341224 11394 raft_consensus.cc:3037] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:23.353741 10015 raft_consensus.cc:2454] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:23.324467 11236 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:23.304133 11539 log.cc:924] Time spent T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.133s user 0.000s sys 0.001s
I20240426 09:55:23.380626 9982 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:23.354861 10089 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
W20240426 09:55:23.291589 11494 log.cc:924] Time spent T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 1.061s user 0.002s sys 0.002s
I20240426 09:55:23.351338 9940 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:23.385035 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:23.386413 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.356019 10087 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.346602 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.298591 11315 raft_consensus.cc:3032] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:23.358094 9979 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:23.360126 10055 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:23.392208 10055 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:23.360548 9934 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:23.292701 11474 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.352586 10090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1022) took 1048 ms. Trace:
I20240426 09:55:23.399181 10090 rpcz_store.cc:276] 0426 09:55:22.304159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:22.304257 (+ 98us) service_pool.cc:225] Handling call
0426 09:55:23.287241 (+982984us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b58000b72b0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2bc20e3 00007f9ca2c25c4a 00007f9ca2c3d53f 00007f9ca2c3ecde 00007f9ca2c4a3b8 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c
0426 09:55:23.352532 (+ 65291us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1394712320}
I20240426 09:55:23.400153 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:23.367614 10056 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:23.367539 9933 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
W20240426 09:55:23.371773 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 151536us
I20240426 09:55:23.362711 11418 raft_consensus.cc:2781] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:23.387070 10015 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
W20240426 09:55:23.358069 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 105208us
I20240426 09:55:23.354461 11536 raft_consensus.cc:686] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:23.316377 11524 raft_consensus.cc:2781] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:23.459306 11435 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:23.379897 10611 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:23.459985 11506 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:23.396221 11463 raft_consensus.cc:2726] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:23.463874 11253 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.396942 10009 raft_consensus.cc:2379] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:23.366297 11428 raft_consensus.cc:729] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:23.398598 9979 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:23.380048 11375 raft_consensus.cc:2726] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:23.467563 9934 raft_consensus.cc:1261] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20240426 09:55:23.467932 9934 pending_rounds.cc:76] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9: Aborting all ops after (but not including) 1
I20240426 09:55:23.468241 9934 pending_rounds.cc:97] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240426 09:55:23.468487 9934 raft_consensus.cc:2866] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:23.469949 9936 raft_consensus.cc:1226] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:23.388948 11423 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:23.459909 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:23.466184 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 67672.4us
I20240426 09:55:23.402143 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:23.472193 9979 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:23.472916 9936 raft_consensus.cc:1226] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:23.475234 11536 consensus_queue.cc:237] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:23.476745 9932 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:23.477943 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:23.478550 11423 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:23.479581 9939 raft_consensus.cc:1226] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:23.479965 11517 raft_consensus.cc:2781] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:23.480365 11517 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:23.480527 11423 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:23.480743 11517 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.437407 10055 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:23.391083 11315 raft_consensus.cc:729] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
W20240426 09:55:23.482053 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 44740.7us
I20240426 09:55:23.396365 11411 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:23.485922 11411 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:23.486295 11411 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:23.487331 11320 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.487959 10620 raft_consensus.cc:1261] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:23.466291 11428 consensus_queue.cc:260] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.488353 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:23.488857 11428 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:23.443384 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:23.447196 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:23.386252 11356 consensus_queue.cc:1046] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20240426 09:55:23.445614 11419 raft_consensus.cc:2781] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:23.452529 11418 raft_consensus.cc:686] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [term 4 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:23.384399 11236 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.453510 10618 raft_consensus.cc:1261] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 09:55:23.455511 10089 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:23.455935 11521 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.379669 11452 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.470793 11524 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:23.482124 10088 raft_consensus.cc:2454] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
I20240426 09:55:23.529407 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:23.530069 10088 raft_consensus.cc:2454] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:23.384482 11508 raft_consensus.cc:2726] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:23.446259 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:23.388875 11546 raft_consensus.cc:2781] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:23.467869 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:23.499840 11375 raft_consensus.cc:2781] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:23.481544 9939 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:23.436954 11543 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:23.481515 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 900) took 1019 ms. Trace:
I20240426 09:55:23.465750 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.401660 10056 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:23.364079 11473 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.477794 9932 raft_consensus.cc:1226] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:23.562194 10009 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:23.510129 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:23.393468 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:23.563803 9933 raft_consensus.cc:2379] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:23.560278 10008 rpcz_store.cc:276] 0426 09:55:22.461892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:22.468297 (+ 6405us) service_pool.cc:225] Handling call
0426 09:55:22.501626 (+ 33329us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:22.855025 (+353399us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:23.481266 (+626241us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x7b54000569d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:23.481440 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":667058560}
I20240426 09:55:23.565431 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.567193 11315 consensus_queue.cc:260] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.568250 11315 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:23.563287 10082 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:23.569589 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:23.570353 10082 raft_consensus.cc:2379] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 2: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:23.571422 11546 raft_consensus.cc:686] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:23.571918 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.572568 10009 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:23.477623 11416 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:23.493855 10087 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:23.482642 11415 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.525310 9979 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:23.483204 9906 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:23.540454 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:23.503819 11549 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:23.503984 11419 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
W20240426 09:55:23.560895 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 159286us
W20240426 09:55:23.591181 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 59742.7us
I20240426 09:55:23.577576 10090 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:23.537559 11524 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.591794 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:23.593297 10056 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:23.562292 11508 raft_consensus.cc:2781] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:23.550622 11431 raft_consensus.cc:3032] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Stepping down as leader of term 2
W20240426 09:55:23.590229 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 107087us
I20240426 09:55:23.629834 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:23.573014 9937 raft_consensus.cc:2379] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:23.593602 10583 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:23.571211 9932 raft_consensus.cc:1226] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:23.566617 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:23.549503 11375 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:23.625746 11508 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 09:55:23.628293 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 103075us
I20240426 09:55:23.649667 9906 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:23.634160 10583 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:23.632915 11381 raft_consensus.cc:2781] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:23.602087 11419 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:23.601028 11337 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.606431 11320 raft_consensus.cc:2781] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:23.627127 10056 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:23.562409 11350 consensus_queue.cc:1035] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.075s
I20240426 09:55:23.643622 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:23.630231 11427 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:23.560103 11543 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:23.632793 11431 raft_consensus.cc:729] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:23.631347 11435 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:23.634438 11457 raft_consensus.cc:2781] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:23.561018 11423 consensus_queue.cc:1035] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.083s
I20240426 09:55:23.645072 11375 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.572381 11546 consensus_queue.cc:237] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.641618 11401 raft_consensus.cc:2781] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:23.639632 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:23.660159 9939 raft_consensus.cc:2354] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 2. Current term is 3.
I20240426 09:55:23.666989 11463 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election won for term 3
W20240426 09:55:23.653494 11405 consensus_meta.cc:287] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.614s user 0.014s sys 0.003s
I20240426 09:55:23.644642 11454 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:23.642343 11545 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:23.692001 11452 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.665361 10056 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:23.657380 11513 consensus_queue.cc:1046] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20240426 09:55:23.693321 11513 raft_consensus.cc:3032] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:23.693567 10056 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:23.650707 11508 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:23.693914 11343 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:23.601948 11549 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.655125 11402 raft_consensus.cc:2781] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:23.694550 11473 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.694939 11402 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:23.695079 10083 raft_consensus.cc:2454] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
I20240426 09:55:23.660483 10614 raft_consensus.cc:2379] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 5: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:23.652772 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:23.696056 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:23.667237 11320 raft_consensus.cc:686] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:23.696539 11464 consensus_queue.cc:1046] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:23.697693 11464 raft_consensus.cc:3032] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:23.697413 11320 consensus_queue.cc:237] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.663048 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:23.657557 11461 raft_consensus.cc:2781] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:23.683146 11401 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:23.671154 11457 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:23.691334 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:23.666124 11543 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.654965 10008 raft_consensus.cc:2354] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 2. Current term is 3.
I20240426 09:55:23.694391 11343 raft_consensus.cc:686] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:23.702251 11405 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.666245 11381 raft_consensus.cc:686] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:23.695999 10009 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:23.671499 11490 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.668592 11427 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.764037 11454 raft_consensus.cc:2726] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:23.764595 11472 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:23.671244 11506 consensus_queue.cc:1035] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.096s
I20240426 09:55:23.695362 11402 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:23.641966 11418 consensus_queue.cc:237] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:23.691236 11521 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:23.770399 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 62599.2us
I20240426 09:55:23.686952 11404 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.822655 11404 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:23.692143 11463 raft_consensus.cc:686] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:23.823587 11463 consensus_queue.cc:237] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.699723 11549 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.704206 11461 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:23.703513 11536 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:23.769150 11401 raft_consensus.cc:3037] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:23.723688 11513 raft_consensus.cc:729] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:23.769342 11457 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.790082 10013 raft_consensus.cc:2379] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 5: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:23.789011 11548 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:23.698040 11464 raft_consensus.cc:729] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:23.827721 10055 leader_election.cc:398] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 2. Current term is 3.
I20240426 09:55:23.849293 10088 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:23.849586 10055 leader_election.cc:401] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 09:55:23.782011 11236 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.831153 11427 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:23.831583 11430 consensus_meta.cc:287] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.575s user 0.005s sys 0.008s
I20240426 09:55:23.828397 11461 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:23.792343 9906 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:23.839891 11394 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:23.843802 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:23.776434 11431 consensus_queue.cc:260] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.850900 10056 leader_election.cc:398] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 election: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 2. Current term is 3.
I20240426 09:55:23.778889 11411 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.899415 9979 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:23.827538 11416 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.865984 11428 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:55:23.868242 11542 log.cc:924] Time spent T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.691s user 0.001s sys 0.000s
I20240426 09:55:23.908079 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1232) took 2460 ms. Trace:
I20240426 09:55:23.872094 11464 consensus_queue.cc:260] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.873708 11424 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:23.874190 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1206) took 1017 ms. Trace:
I20240426 09:55:23.875025 10016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 923) took 1105 ms. Trace:
I20240426 09:55:23.876916 11504 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:23.877758 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:23.848047 11508 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:23.875828 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 83571us
I20240426 09:55:23.835114 11549 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:23.903993 11411 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:23.904549 10090 raft_consensus.cc:2379] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 4: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
W20240426 09:55:23.905532 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 536330us
I20240426 09:55:23.906807 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:23.864439 11343 consensus_queue.cc:237] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.907347 11490 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:23.908143 11517 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.908779 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:23.908578 11419 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:23.912475 11545 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:23.867936 11381 consensus_queue.cc:237] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:23.916949 10007 rpcz_store.cc:276] 0426 09:55:21.447985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:21.448380 (+ 395us) service_pool.cc:225] Handling call
0426 09:55:21.448654 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:22.281751 (+833097us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:23.907801 (+1626050us) spinlock_profiling.cc:243] Waited 810 ms on lock 0x7b540025ec54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:23.907997 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1862924032}
I20240426 09:55:23.865602 11430 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.861724 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.000867 10014 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
W20240426 09:55:23.969756 10053 service_pool.cc:135] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:59746 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 1 similar messages]
I20240426 09:55:23.990209 11461 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.973731 10054 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:23.973548 11504 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:23.971971 9904 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:23.862905 11548 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:24.106344 11548 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:24.106966 11464 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
W20240426 09:55:24.107128 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 135231us
I20240426 09:55:24.108762 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:23.917663 10009 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.980901 11405 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.112622 11473 raft_consensus.cc:2781] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:24.113014 11473 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:24.113338 11473 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.974445 10016 rpcz_store.cc:276] 0426 09:55:22.769181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:23.436293 (+667112us) service_pool.cc:225] Handling call
0426 09:55:23.436619 (+ 326us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:23.436818 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:23.436841 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:23.447834 (+ 10993us) log.cc:842] Serialized 26 byte log entry
0426 09:55:23.636594 (+188760us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:23.636628 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:23.636695 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:23.874828 (+238133us) raft_consensus.cc:1671] finished
0426 09:55:23.874846 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:23.874973 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":188227,"threads_started":1,"wal-append.queue_time_us":329318}
I20240426 09:55:23.980662 10612 raft_consensus.cc:2379] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:23.981869 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1029) took 1193 ms. Trace:
I20240426 09:55:24.117342 10085 rpcz_store.cc:276] 0426 09:55:22.784273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:22.784946 (+ 673us) service_pool.cc:225] Handling call
0426 09:55:23.906629 (+1121683us) spinlock_profiling.cc:243] Waited 3.46 ms on lock 0x7b58000b0498. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:55:23.977377 (+ 70748us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7964544}
I20240426 09:55:24.120172 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:24.121730 10016 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:24.124071 10054 leader_election.cc:302] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
W20240426 09:55:23.874653 11505 log.cc:924] Time spent T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 1.464s user 0.008s sys 0.002s
I20240426 09:55:23.984812 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:24.133015 10007 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:23.989693 9935 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:23.991088 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:23.992395 11419 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:23.993711 10013 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 917) took 1294 ms. Trace:
I20240426 09:55:23.995160 11545 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.995716 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:23.997671 9940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1007) took 1049 ms. Trace:
I20240426 09:55:23.967605 11431 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:23.835445 11536 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.945250 11427 raft_consensus.cc:2781] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:23.974074 10087 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:23.971179 11315 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:23.971201 11253 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.863143 11472 raft_consensus.cc:2726] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 2. Current term is 3.
I20240426 09:55:24.003118 10008 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:24.092686 10083 raft_consensus.cc:2379] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 4: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:23.918195 11402 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:24.094704 11411 raft_consensus.cc:2781] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:24.095197 11375 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.977161 9934 rpcz_store.cc:276] 0426 09:55:22.857191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:23.469420 (+612229us) service_pool.cc:225] Handling call
0426 09:55:23.469776 (+ 356us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:23.482576 (+ 12800us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:23.500286 (+ 17710us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:23.600395 (+100109us) log.cc:842] Serialized 26 byte log entry
0426 09:55:23.794877 (+194482us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:23.829939 (+ 35062us) log.cc:842] Serialized 14 byte log entry
0426 09:55:23.830155 (+ 216us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:23.830230 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:23.873935 (+ 43705us) raft_consensus.cc:1671] finished
0426 09:55:23.873967 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:23.874135 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":10673,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":2567,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":177,"spinlock_wait_cycles":21376,"thread_start_us":132008,"threads_started":1,"wal-append.queue_time_us":211188}
I20240426 09:55:24.104291 11504 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:23.977367 11549 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.227856 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:24.229348 11549 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.131547 10089 raft_consensus.cc:2379] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
W20240426 09:55:24.141691 10086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1035) took 1321 ms (client timeout 1654 ms). Trace:
I20240426 09:55:24.139801 10007 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:24.133601 11350 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:55:24.230551 10086 rpcz_store.cc:269] 0426 09:55:22.820716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:22.825667 (+ 4951us) service_pool.cc:225] Handling call
0426 09:55:24.141508 (+1315841us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b5400181ad4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:24.141638 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":276795264}
W20240426 09:55:24.153803 9935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 993) took 1580 ms (client timeout 1532 ms). Trace:
I20240426 09:55:24.150537 10013 rpcz_store.cc:276] 0426 09:55:22.699386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:23.058964 (+359578us) service_pool.cc:225] Handling call
0426 09:55:23.993538 (+934574us) spinlock_profiling.cc:243] Waited 713 ms on lock 0x7b5400049f54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:23.993659 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1640776448}
W20240426 09:55:24.231457 9935 rpcz_store.cc:269] 0426 09:55:22.558845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:22.559053 (+ 208us) service_pool.cc:225] Handling call
0426 09:55:23.908643 (+1349590us) spinlock_profiling.cc:243] Waited 1.62 ms on lock 0x7b5800004598. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:55:24.138955 (+230312us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3721600}
I20240426 09:55:24.232216 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.001821 11407 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:24.233045 10013 raft_consensus.cc:2396] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 09:55:24.231215 11350 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.233906 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:23.931123 11428 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:23.981439 11490 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:24.235069 11565 log.cc:924] Time spent T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.237s user 0.001s sys 0.000s
I20240426 09:55:24.153671 11404 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:24.155582 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:24.236802 11404 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.237322 10012 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:24.238683 11404 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:24.132596 11408 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:24.239346 11408 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.150036 11462 raft_consensus.cc:2726] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:24.150321 11405 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:24.241335 11405 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.134449 11435 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:24.242339 11435 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.150810 11452 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:24.244144 11452 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:24.244884 9934 raft_consensus.cc:1226] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 09:55:24.198093 11548 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:24.245867 11548 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.154664 11506 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:24.227209 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 137) took 2172 ms. Trace:
I20240426 09:55:24.247291 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:24.247501 10011 rpcz_store.cc:276] 0426 09:55:22.055006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:22.055158 (+ 152us) service_pool.cc:225] Handling call
0426 09:55:22.099386 (+ 44228us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:22.200313 (+100927us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:22.200340 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:22.200826 (+ 486us) log.cc:842] Serialized 26 byte log entry
0426 09:55:22.303197 (+102371us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:22.303240 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:22.303306 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:23.874171 (+1570865us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:55:24.226907 (+352736us) raft_consensus.cc:1671] finished
0426 09:55:24.226931 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.227151 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":18477,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":13971,"spinlock_wait_cycles":25117056,"thread_start_us":116955,"threads_started":2,"wal-append.queue_time_us":186442}
I20240426 09:55:24.209141 10016 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:24.227614 11411 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:24.229264 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:24.151510 9940 rpcz_store.cc:276] 0426 09:55:22.948676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:23.506919 (+558243us) service_pool.cc:225] Handling call
0426 09:55:23.541206 (+ 34287us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:23.541540 (+ 334us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:23.541567 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:23.608254 (+ 66687us) log.cc:842] Serialized 26 byte log entry
0426 09:55:23.788983 (+180729us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:23.789022 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:23.789083 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:23.997474 (+208391us) raft_consensus.cc:1671] finished
0426 09:55:23.997496 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:23.997620 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7844,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":15637,"thread_start_us":145175,"threads_started":1,"wal-append.queue_time_us":267811}
I20240426 09:55:23.925827 11513 consensus_queue.cc:260] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.236531 9939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1221) took 1204 ms. Trace:
I20240426 09:55:24.209825 11427 raft_consensus.cc:686] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:24.213476 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 144) took 1149 ms. Trace:
I20240426 09:55:24.253033 10007 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 4
I20240426 09:55:24.264348 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.277357 10011 raft_consensus.cc:2396] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 09:55:24.247292 11506 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.225176 9932 raft_consensus.cc:2379] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:24.235117 11350 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.273020 11435 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.297945 10007 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:24.273355 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:24.213142 11253 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:24.236403 11428 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:24.267810 11411 raft_consensus.cc:3037] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:24.209823 11315 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:24.278645 11402 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.279300 11375 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:24.279639 11557 log.cc:924] Time spent T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.430s user 0.006s sys 0.005s
W20240426 09:55:24.280113 11524 consensus_meta.cc:287] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.649s user 0.008s sys 0.000s
I20240426 09:55:24.280391 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:55:24.280056 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 893290us
I20240426 09:55:24.312343 9982 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:24.230741 11549 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:24.271492 9939 rpcz_store.cc:276] 0426 09:55:23.032627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:23.688573 (+655946us) service_pool.cc:225] Handling call
0426 09:55:23.688897 (+ 324us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:23.689210 (+ 313us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:23.689239 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:23.689661 (+ 422us) log.cc:842] Serialized 26 byte log entry
0426 09:55:23.857019 (+167358us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:23.888200 (+ 31181us) log.cc:842] Serialized 14 byte log entry
0426 09:55:23.897996 (+ 9796us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:23.898067 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:24.236346 (+338279us) raft_consensus.cc:1671] finished
0426 09:55:24.236364 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.236481 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9573,"prepare.queue_time_us":112200,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":2960,"raft.queue_time_us":11048,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":16144,"thread_start_us":167055,"threads_started":1,"wal-append.queue_time_us":286008}
I20240426 09:55:24.249632 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:24.316776 10016 raft_consensus.cc:2379] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:24.317541 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 135) took 2445 ms. Trace:
I20240426 09:55:24.236482 11490 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:23.982820 10055 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:59762 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 897)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 898)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 878)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 872)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 115)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 887)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1053)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 119)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 892)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 871)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 884)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 879)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 890)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 891)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1058)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1059)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 886)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1057)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 106)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 107)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 108)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1038)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1040)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1041)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 114)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1042)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1045)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1046)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1047)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1048)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1051)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 117)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 868)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 869)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 870)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 876)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1054)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 883)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 885)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1055)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 889)
I20240426 09:55:24.292573 11508 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:24.293378 11430 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:24.293774 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1213) took 1434 ms. Trace:
I20240426 09:55:24.263512 10012 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:24.330865 10012 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1251) took 1419 ms. Trace:
I20240426 09:55:24.331378 10012 rpcz_store.cc:276] 0426 09:55:22.911092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.330173 (+1419081us) service_pool.cc:225] Handling call
0426 09:55:24.330435 (+ 262us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.330582 (+ 147us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:24.330604 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:24.330637 (+ 33us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:24.330663 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.330709 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.330813 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:55:24.298419 11521 consensus_queue.cc:1159] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d3a480e831d649139ff016fb68aee3ee" responder_term: 3 status { last_received { term: 3 index: 3 } last_committed_idx: 2 last_received_current_leader { term: 3 index: 3 } }
W20240426 09:55:24.335148 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 182602us
I20240426 09:55:24.329375 9933 rpcz_store.cc:276] 0426 09:55:22.858886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:23.446060 (+587174us) service_pool.cc:225] Handling call
0426 09:55:24.293596 (+847536us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x7b54002378d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:24.293721 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32645504}
I20240426 09:55:24.300143 10007 raft_consensus.cc:1226] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240426 09:55:24.232975 11407 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.301718 9940 raft_consensus.cc:2379] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:24.272588 11394 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.312500 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:24.318082 10016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1250) took 1515 ms. Trace:
I20240426 09:55:24.346700 10012 raft_consensus.cc:1226] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:24.302145 11463 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:24.324136 9982 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:24.348536 10012 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1252) took 1432 ms. Trace:
I20240426 09:55:24.297147 11401 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:24.324641 11408 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.324501 11517 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.347936 10016 rpcz_store.cc:276] 0426 09:55:22.802338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.317353 (+1515015us) service_pool.cc:225] Handling call
0426 09:55:24.317633 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.317789 (+ 156us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:24.317813 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:24.317854 (+ 41us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:24.317884 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.317932 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.318031 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.357429 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.317945 10010 rpcz_store.cc:276] 0426 09:55:21.872314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:21.872434 (+ 120us) service_pool.cc:225] Handling call
0426 09:55:22.853272 (+980838us) spinlock_profiling.cc:243] Waited 5.7 ms on lock 0x7b580004ec98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:55:22.853440 (+ 168us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:22.971689 (+118249us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:22.971716 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:22.971758 (+ 42us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:22.971785 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.317314 (+1345529us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54003be2d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:24.317368 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.317485 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":284652032}
I20240426 09:55:24.322854 11546 raft_consensus.cc:2726] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240426 09:55:24.334944 11560 log.cc:924] Time spent T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.364s user 0.001s sys 0.000s
I20240426 09:55:24.305706 11253 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.369860 10015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 924) took 1600 ms. Trace:
I20240426 09:55:24.275875 9937 rpcz_store.cc:276] 0426 09:55:23.060894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.003055 (+942161us) service_pool.cc:225] Handling call
0426 09:55:24.131798 (+128743us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.197082 (+ 65284us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:24.199049 (+ 1967us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:24.199101 (+ 52us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:24.199150 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.199226 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.210107 (+ 10881us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.318981 11524 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.342093 10013 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 929) took 1464 ms. Trace:
I20240426 09:55:24.381521 10013 rpcz_store.cc:276] 0426 09:55:22.878038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.327038 (+1449000us) service_pool.cc:225] Handling call
0426 09:55:24.341637 (+ 14599us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.341807 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:24.341831 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:24.341870 (+ 39us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:24.341903 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.341950 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.342042 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:55:24.353355 9979 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:59746 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 43 similar messages]
I20240426 09:55:24.332742 11337 consensus_queue.cc:1035] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
I20240426 09:55:24.363519 9939 raft_consensus.cc:1261] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:24.364154 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:24.339370 11461 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:24.365367 10014 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1228) took 2952 ms (client timeout 1635 ms). Trace:
I20240426 09:55:24.368634 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.348870 10012 rpcz_store.cc:276] 0426 09:55:22.916202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.332274 (+1416072us) service_pool.cc:225] Handling call
0426 09:55:24.344494 (+ 12220us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.348331 (+ 3837us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.348489 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.373667 10055 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:24.371201 11422 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:24.370288 10015 rpcz_store.cc:276] 0426 09:55:22.769432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:23.445939 (+676507us) service_pool.cc:225] Handling call
0426 09:55:23.470290 (+ 24351us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:23.470562 (+ 272us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:23.470588 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:23.487936 (+ 17348us) log.cc:842] Serialized 26 byte log entry
0426 09:55:23.673717 (+185781us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:23.673752 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:23.673813 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:24.279124 (+605311us) spinlock_profiling.cc:243] Waited 7.57 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:55:24.356019 (+ 76895us) raft_consensus.cc:1671] finished
0426 09:55:24.356046 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.369794 (+ 13748us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41267,"raft.queue_time_us":1522,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":22688,"spinlock_wait_cycles":17411840,"thread_start_us":185473,"threads_started":1,"wal-append.queue_time_us":296943}
I20240426 09:55:24.360560 11452 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:24.377000 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:24.342950 11548 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.394309 9936 raft_consensus.cc:1226] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 3. Ops: []
I20240426 09:55:24.315820 11549 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:24.400241 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:24.400853 9939 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:24.383592 10618 raft_consensus.cc:3037] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:24.330052 11533 consensus_queue.cc:1046] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20240426 09:55:24.383244 11546 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:24.387876 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.388113 10055 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
W20240426 09:55:24.386881 10014 rpcz_store.cc:269] 0426 09:55:21.402327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:21.402440 (+ 113us) service_pool.cc:225] Handling call
0426 09:55:24.354885 (+2952445us) spinlock_profiling.cc:243] Waited 2.54 s on lock 0x7b54003bd8d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:24.355007 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5841158016}
I20240426 09:55:24.376709 9906 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:24.355172 11430 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:24.365403 11410 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:24.393075 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:24.397126 10016 raft_consensus.cc:2379] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 4: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:24.402432 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1008) took 1453 ms. Trace:
I20240426 09:55:24.391252 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:24.400378 11520 raft_consensus.cc:3032] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:24.412364 11461 raft_consensus.cc:2781] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:24.406481 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:24.412657 11546 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:24.427207 11570 log.cc:924] Time spent T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20240426 09:55:24.426743 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.438823 10088 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:24.439052 11546 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.439759 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.440454 10088 raft_consensus.cc:2379] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:24.441128 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.442215 9941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1209) took 1584 ms. Trace:
I20240426 09:55:24.442567 9941 rpcz_store.cc:276] 0426 09:55:22.857913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:23.471691 (+613778us) service_pool.cc:225] Handling call
0426 09:55:23.508651 (+ 36960us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:23.702330 (+193679us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:23.702362 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:23.702751 (+ 389us) log.cc:842] Serialized 26 byte log entry
0426 09:55:23.861106 (+158355us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:24.335592 (+474486us) log.cc:842] Serialized 14 byte log entry
0426 09:55:24.348966 (+ 13374us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.441826 (+ 92860us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b54003c32d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:24.441887 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:24.441933 (+ 46us) raft_consensus.cc:1671] finished
0426 09:55:24.442048 (+ 115us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.442165 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40745,"prepare.queue_time_us":136522,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":39609,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":73125,"spinlock_wait_cycles":440070912,"thread_start_us":526020,"threads_started":2,"wal-append.queue_time_us":268117}
I20240426 09:55:24.444581 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:24.445611 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:55:24.447167 11568 log.cc:924] Time spent T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
I20240426 09:55:24.447897 11533 raft_consensus.cc:3032] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Stepping down as leader of term 2
W20240426 09:55:24.448351 11569 log.cc:924] Time spent T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20240426 09:55:24.449218 11315 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:24.449443 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:24.417141 11430 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.450625 11401 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.429339 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:24.431231 11543 consensus_meta.cc:287] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.650s user 0.006s sys 0.002s
I20240426 09:55:24.452487 11543 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.452857 11524 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.454360 11543 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.395038 11422 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:24.434095 10008 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:24.450660 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1211) took 1592 ms. Trace:
I20240426 09:55:24.428799 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:24.446554 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:24.379916 11381 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:24.334076 11427 consensus_queue.cc:237] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.386130 11454 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:24.367630 11457 consensus_meta.cc:287] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.529s user 0.009s sys 0.000s
I20240426 09:55:24.387115 11517 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:24.407563 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:24.408344 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.354964 11463 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.387081 11536 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.409349 9906 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:24.392175 11336 raft_consensus.cc:2781] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:24.412420 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.415017 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.415297 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:24.415655 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:24.351824 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 928) took 1565 ms. Trace:
I20240426 09:55:24.478950 10007 rpcz_store.cc:276] 0426 09:55:22.786175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.299525 (+1513350us) service_pool.cc:225] Handling call
0426 09:55:24.299899 (+ 374us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:24.350811 (+ 50912us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.351768 (+ 957us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.414925 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:24.480293 9939 raft_consensus.cc:2354] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
I20240426 09:55:24.481087 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:24.482178 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:24.403008 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.396514 11405 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.422101 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.485853 10083 raft_consensus.cc:2454] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:24.486395 10083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 872) took 1028 ms. Trace:
I20240426 09:55:24.422714 9934 raft_consensus.cc:2379] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:24.423480 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.424247 10009 raft_consensus.cc:2454] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:24.416179 10084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1027) took 1628 ms. Trace:
W20240426 09:55:24.416090 11564 log.cc:924] Time spent T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
I20240426 09:55:24.458321 11506 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:24.448436 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 629132us
I20240426 09:55:24.458176 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:24.414155 9938 rpcz_store.cc:276] 0426 09:55:22.948947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:23.538324 (+589377us) service_pool.cc:225] Handling call
0426 09:55:23.538641 (+ 317us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:23.538996 (+ 355us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:23.539026 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:23.539431 (+ 405us) log.cc:842] Serialized 26 byte log entry
0426 09:55:23.779461 (+240030us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:23.779502 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:23.779571 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:24.402236 (+622665us) raft_consensus.cc:1671] finished
0426 09:55:24.402254 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.402381 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":101,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":84,"thread_start_us":239754,"threads_started":1,"wal-append.queue_time_us":336505}
I20240426 09:55:24.502369 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 144) took 1530 ms. Trace:
I20240426 09:55:24.507282 10007 rpcz_store.cc:276] 0426 09:55:22.972009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.479760 (+1507751us) service_pool.cc:225] Handling call
0426 09:55:24.501839 (+ 22079us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.502036 (+ 197us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:24.502061 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:24.502105 (+ 44us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:24.502135 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.502187 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.502307 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.459129 9982 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:24.457020 9936 rpcz_store.cc:276] 0426 09:55:22.858388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:23.473580 (+615192us) service_pool.cc:225] Handling call
0426 09:55:23.473821 (+ 241us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.397750 (+923929us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.450461 (+ 52711us) spinlock_profiling.cc:243] Waited 920 ms on lock 0x7b54002616d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:24.450609 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2116567936}
W20240426 09:55:24.427614 11567 log.cc:924] Time spent T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20240426 09:55:24.464375 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:24.462597 11422 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:24.512984 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:24.435595 11461 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:24.418387 11424 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:24.448308 11533 raft_consensus.cc:729] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:24.437750 10612 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45570 (request call id 153) took 1179 ms. Trace:
I20240426 09:55:24.436570 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:24.435312 11520 raft_consensus.cc:729] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:24.510919 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:55:24.510003 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 100715us
I20240426 09:55:24.511467 10013 raft_consensus.cc:2379] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:24.512187 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:24.534898 9906 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:24.491791 11549 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:24.473894 10015 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:24.513409 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:24.512902 10087 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:24.542591 9941 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
W20240426 09:55:24.516247 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 57206.7us
I20240426 09:55:24.543727 9982 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:24.545126 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:24.520087 10014 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 145) took 1517 ms. Trace:
I20240426 09:55:24.545941 10014 rpcz_store.cc:276] 0426 09:55:23.002282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.494811 (+1492529us) service_pool.cc:225] Handling call
0426 09:55:24.519630 (+ 24819us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.519802 (+ 172us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:24.519825 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:24.519863 (+ 38us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:55:24.519892 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.519938 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.520036 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.522419 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:24.523639 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 170666us
I20240426 09:55:24.487030 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:24.536945 9938 raft_consensus.cc:2354] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3.
I20240426 09:55:24.538004 9939 raft_consensus.cc:1226] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:24.511991 11452 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:24.471745 11546 consensus_queue.cc:1035] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20240426 09:55:24.548290 9979 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:24.516690 11461 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:24.551894 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.514873 11328 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:24.515549 11336 raft_consensus.cc:686] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:24.515663 10082 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:55:24.535547 11405 raft_consensus.cc:2726] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:24.476886 11457 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:24.521255 11508 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:24.571905 11508 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.573634 11508 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:24.574234 11457 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:24.574447 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:24.570667 10082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 115) took 1217 ms. Trace:
I20240426 09:55:24.575065 10616 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:24.575182 10082 rpcz_store.cc:276] 0426 09:55:23.352921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.423235 (+1070314us) service_pool.cc:225] Handling call
0426 09:55:24.570612 (+147377us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.478093 11473 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:24.486712 10083 rpcz_store.cc:276] 0426 09:55:23.457867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.421852 (+963985us) service_pool.cc:225] Handling call
0426 09:55:24.486335 (+ 64483us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.521459 11375 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:24.581084 9940 raft_consensus.cc:1261] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:24.470724 11381 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.563803 10055 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:48202 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 119)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 892)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 871)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 884)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 879)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 890)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 121)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 891)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1058)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 120)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1059)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1062)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1065)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1064)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1063)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 123)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 886)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1057)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 106)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 107)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 108)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1038)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1040)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1041)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 114)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1042)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1045)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1046)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1047)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1048)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1051)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 117)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 868)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 869)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 870)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 876)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1054)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 883)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 885)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1055)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 889)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1060)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1061)
I20240426 09:55:24.543895 9937 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:24.566722 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:24.567880 10015 raft_consensus.cc:1261] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 09:55:24.552306 11462 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:24.569221 11463 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:24.521360 10081 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:24.567836 11328 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.569846 10586 leader_election.cc:398] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3.
I20240426 09:55:24.557760 11464 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:24.518739 10612 rpcz_store.cc:276] 0426 09:55:23.271130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:23.276158 (+ 5028us) service_pool.cc:225] Handling call
0426 09:55:24.437553 (+1161395us) spinlock_profiling.cc:243] Waited 639 ms on lock 0x7b54006004d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:24.437680 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1470320000}
I20240426 09:55:24.478279 11517 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:24.473927 11454 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.575223 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:24.571007 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 925) took 1801 ms. Trace:
I20240426 09:55:24.581346 11452 raft_consensus.cc:686] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:24.597616 10011 rpcz_store.cc:276] 0426 09:55:22.769727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.291275 (+1521548us) service_pool.cc:225] Handling call
0426 09:55:24.306871 (+ 15596us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:24.307212 (+ 341us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:24.307240 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:24.310408 (+ 3168us) log.cc:842] Serialized 26 byte log entry
0426 09:55:24.462871 (+152463us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:24.462906 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.462964 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:24.570801 (+107837us) raft_consensus.cc:1671] finished
0426 09:55:24.570819 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.570953 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":88,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":4532,"thread_start_us":132998,"threads_started":1,"wal-append.queue_time_us":219454}
I20240426 09:55:24.002254 10053 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:59746 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 898)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 878)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 872)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 115)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 887)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1053)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 119)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 892)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 871)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 884)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 879)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 890)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 891)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1058)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1059)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 886)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1057)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 106)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 107)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 108)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1038)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1040)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1041)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 114)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1042)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1045)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1046)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1047)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1048)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1051)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 117)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 868)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 869)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 870)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 876)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1054)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 883)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 885)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1055)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 889)
I20240426 09:55:24.599373 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 937) took 1097 ms. Trace:
I20240426 09:55:24.599777 10011 rpcz_store.cc:276] 0426 09:55:23.501947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.598676 (+1096729us) service_pool.cc:225] Handling call
0426 09:55:24.598960 (+ 284us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.599107 (+ 147us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:24.599132 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:24.599165 (+ 33us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:24.599191 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.599238 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.599329 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.601094 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 148) took 1072 ms. Trace:
I20240426 09:55:24.601281 10011 rpcz_store.cc:276] 0426 09:55:23.528771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.600563 (+1071792us) service_pool.cc:225] Handling call
0426 09:55:24.600806 (+ 243us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.600918 (+ 112us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:24.600931 (+ 13us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:24.600955 (+ 24us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:24.600971 (+ 16us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.601000 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.601063 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.564451 10087 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:24.543920 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.503875 10084 rpcz_store.cc:276] 0426 09:55:22.783741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:22.791004 (+ 7263us) service_pool.cc:225] Handling call
0426 09:55:24.347289 (+1556285us) spinlock_profiling.cc:243] Waited 3.2 ms on lock 0x7b58000b0498. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:55:24.365101 (+ 17812us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.383215 (+ 18114us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:24.386535 (+ 3320us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:24.391789 (+ 5254us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:55:24.394530 (+ 2741us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.401132 (+ 6602us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.412048 (+ 10916us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7370112}
I20240426 09:55:24.533542 11424 raft_consensus.cc:686] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:24.580864 10086 raft_consensus.cc:2379] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:24.590647 11490 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:24.589488 11430 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:24.575320 9835 catalog_manager.cc:5484] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:24.552488 11422 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:24.568454 11336 consensus_queue.cc:237] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.586593 10015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 147) took 1131 ms. Trace:
I20240426 09:55:24.570572 11407 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.590170 9906 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:24.590670 9933 raft_consensus.cc:1226] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3. Ops: []
W20240426 09:55:24.591077 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 167436us
W20240426 09:55:24.604981 11576 log.cc:924] Time spent T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20240426 09:55:24.594380 10586 leader_election.cc:401] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 09:55:24.579735 11435 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:24.558785 11582 raft_consensus.cc:2781] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:24.580299 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1253) took 1654 ms. Trace:
I20240426 09:55:24.602614 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1053) took 1203 ms. Trace:
I20240426 09:55:24.580947 10014 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 933) took 1085 ms. Trace:
I20240426 09:55:24.613008 10015 rpcz_store.cc:276] 0426 09:55:23.455071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.550788 (+1095717us) service_pool.cc:225] Handling call
0426 09:55:24.567597 (+ 16809us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.586349 (+ 18752us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.586541 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.687171 10014 rpcz_store.cc:276] 0426 09:55:23.495678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.550059 (+1054381us) service_pool.cc:225] Handling call
0426 09:55:24.580485 (+ 30426us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.580655 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:24.580682 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:24.580723 (+ 41us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:24.580752 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.580801 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.580893 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.687705 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 147) took 1364 ms. Trace:
I20240426 09:55:24.688074 9935 rpcz_store.cc:276] 0426 09:55:23.323824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.232366 (+908542us) service_pool.cc:225] Handling call
0426 09:55:24.232626 (+ 260us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.417887 (+185261us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:24.417917 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:24.417955 (+ 38us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:55:24.417983 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.687497 (+269514us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b540019c154. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:24.687541 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.687656 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":425679744}
I20240426 09:55:24.612859 10053 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:59762 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 879)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 890)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 121)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 891)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1058)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 120)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1059)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1062)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1065)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1064)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1066)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1063)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 123)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 886)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1057)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 106)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 107)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 108)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1038)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1040)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1041)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 114)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1042)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1045)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1046)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1047)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1048)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1051)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 117)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 868)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 869)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 870)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 876)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1054)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 883)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 885)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1055)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 889)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1060)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1061)
I20240426 09:55:24.617350 11411 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:24.617900 10054 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
W20240426 09:55:24.695029 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 77204.7us
I20240426 09:55:24.612110 11463 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:24.588440 11533 consensus_queue.cc:260] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.585927 11375 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.598701 11464 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:24.699894 11417 consensus_queue.cc:1035] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:24.700582 11464 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:24.701083 11513 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:55:24.701790 10081 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1028) took 1917 ms (client timeout 2023 ms). Trace:
I20240426 09:55:24.702566 11375 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:24.605780 11315 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:24.599334 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.619632 10009 raft_consensus.cc:1261] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 09:55:24.603320 11404 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:24.621551 10618 raft_consensus.cc:2454] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:24.623018 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.623905 11328 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.611438 11457 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:24.625515 11473 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:24.613070 11422 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:24.705351 11315 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.594748 11444 consensus_queue.cc:1046] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20240426 09:55:24.709008 11444 raft_consensus.cc:3032] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:24.709345 11444 raft_consensus.cc:729] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:24.709419 11253 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.710126 11444 consensus_queue.cc:260] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.710470 11401 consensus_queue.cc:1035] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:24.710878 11517 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:24.626837 11490 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.687196 10008 rpcz_store.cc:276] 0426 09:55:22.925588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.456292 (+1530704us) service_pool.cc:225] Handling call
0426 09:55:24.509356 (+ 53064us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.518980 (+ 9624us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:24.521743 (+ 2763us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:24.532954 (+ 11211us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:24.544009 (+ 11055us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.551921 (+ 7912us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.576700 (+ 24779us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.711829 10012 raft_consensus.cc:1261] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 09:55:24.707763 11457 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.712397 10012 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 930) took 1738 ms. Trace:
I20240426 09:55:24.712771 10012 rpcz_store.cc:276] 0426 09:55:22.973868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.484943 (+1511075us) service_pool.cc:225] Handling call
0426 09:55:24.711366 (+226423us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b580004ec98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:55:24.711518 (+ 152us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.712184 (+ 666us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.712350 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40345856}
I20240426 09:55:24.713729 11457 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:24.707237 11394 raft_consensus.cc:2726] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3.
I20240426 09:55:24.628577 11545 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.594602 11410 raft_consensus.cc:686] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:24.576215 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:24.635144 9982 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
W20240426 09:55:24.638182 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 48086.6us
W20240426 09:55:24.638914 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 69204.6us
I20240426 09:55:24.571043 11520 consensus_queue.cc:260] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.671878 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1260) took 1609 ms. Trace:
I20240426 09:55:24.704290 11336 raft_consensus.cc:3037] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:24.707051 10083 raft_consensus.cc:2379] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 4: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:24.714187 11533 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:24.593199 11462 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.676628 10088 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 887) took 1030 ms. Trace:
I20240426 09:55:24.677453 10086 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 878) took 1194 ms. Trace:
I20240426 09:55:24.677752 10616 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:24.726663 10086 rpcz_store.cc:276] 0426 09:55:23.483068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.429119 (+946051us) service_pool.cc:225] Handling call
0426 09:55:24.677289 (+248170us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b5400083754. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:24.677403 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":289053056}
I20240426 09:55:24.685066 11435 raft_consensus.cc:686] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:24.679159 10090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 881) took 1179 ms. Trace:
I20240426 09:55:24.689761 10053 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:59746 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 879)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 890)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 121)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 891)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1058)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 120)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1059)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1062)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1065)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1064)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1066)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1063)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 123)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 906)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 886)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1057)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 106)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 107)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 108)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1038)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1040)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1041)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 114)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1042)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1045)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1046)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1047)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1048)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1051)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 117)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 868)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 869)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 870)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 876)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1054)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 883)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 885)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1055)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 889)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1060)
I20240426 09:55:24.686583 10087 rpcz_store.cc:276] 0426 09:55:23.399550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.457625 (+1058075us) service_pool.cc:225] Handling call
0426 09:55:24.602569 (+144944us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.661571 11524 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:24.629487 11452 consensus_queue.cc:237] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.621565 11504 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.697985 10614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45564 (request call id 196) took 1668 ms. Trace:
I20240426 09:55:24.617220 11424 consensus_queue.cc:237] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.605110 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:24.702193 10081 rpcz_store.cc:269] 0426 09:55:22.784005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:22.791118 (+ 7113us) service_pool.cc:225] Handling call
0426 09:55:23.990913 (+1199795us) spinlock_profiling.cc:243] Waited 3.16 ms on lock 0x7b58000b0498. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:55:24.701638 (+710725us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b54001ee1d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:24.701736 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":638652928}
I20240426 09:55:24.746867 11416 consensus_queue.cc:1046] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:24.747295 10084 raft_consensus.cc:2454] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:24.703922 10015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34362 (request call id 710) took 1823 ms. Trace:
I20240426 09:55:24.747732 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:24.705834 10085 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:24.708376 11422 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:24.748138 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:24.725387 11230 raft_consensus.cc:2726] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:24.744076 11343 consensus_queue.cc:1046] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20240426 09:55:24.749671 11343 consensus_queue.cc:1225] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Peer f12521b6d51242629cf4297864df00b9 log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20240426 09:55:24.748521 10081 raft_consensus.cc:2354] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3.
I20240426 09:55:24.750175 11490 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.750952 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:24.747886 10614 rpcz_store.cc:276] 0426 09:55:23.029368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:23.031250 (+ 1882us) service_pool.cc:225] Handling call
0426 09:55:24.685730 (+1654480us) spinlock_profiling.cc:243] Waited 557 ms on lock 0x7b540064ee54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:24.697297 (+ 11567us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1280025856}
I20240426 09:55:24.748211 10015 rpcz_store.cc:276] 0426 09:55:22.880778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.696906 (+1816128us) service_pool.cc:225] Handling call
0426 09:55:24.703870 (+ 6964us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.752015 10081 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:24.752190 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:24.752621 11461 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.753532 11504 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.747887 11416 raft_consensus.cc:3032] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:24.748328 10084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 884) took 1121 ms. Trace:
I20240426 09:55:24.754315 11416 raft_consensus.cc:729] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:24.755590 11416 consensus_queue.cc:260] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:24.756733 11416 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:24.754717 10084 rpcz_store.cc:276] 0426 09:55:23.626727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.604872 (+978145us) service_pool.cc:225] Handling call
0426 09:55:24.748272 (+143400us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.759723 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.696198 11582 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:24.720271 10007 rpcz_store.cc:276] 0426 09:55:23.062155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.508209 (+1446054us) service_pool.cc:225] Handling call
0426 09:55:24.508528 (+ 319us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.593811 (+ 85283us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:24.593839 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:24.593880 (+ 41us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:24.593913 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:24.671657 (+ 77744us) spinlock_profiling.cc:243] Waited 85.1 ms on lock 0x7b54003bced4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:24.671698 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:24.671826 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":195744512}
I20240426 09:55:24.760293 10084 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:24.754741 9932 raft_consensus.cc:2379] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:24.761262 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:24.722138 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 87085.2us
I20240426 09:55:24.761816 11381 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.762149 10084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 866) took 1696 ms. Trace:
I20240426 09:55:24.722586 10082 raft_consensus.cc:2379] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 5: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:24.723065 11410 consensus_queue.cc:237] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.719909 9906 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:55:24.763377 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 43537us
I20240426 09:55:24.763656 9982 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:24.763399 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:24.764623 11504 raft_consensus.cc:2781] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:24.764830 9982 leader_election.cc:398] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 pre-election: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
I20240426 09:55:24.765174 11504 raft_consensus.cc:686] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:24.766330 11230 raft_consensus.cc:2726] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:24.766003 11504 consensus_queue.cc:237] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.716188 9941 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:24.766058 9982 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:24.768981 9906 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:24.769742 11524 raft_consensus.cc:2781] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:24.770148 11524 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:24.770493 11524 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:24.769651 11407 raft_consensus.cc:2726] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 09:55:24.771924 11590 log.cc:924] Time spent T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.073s user 0.002s sys 0.000s
I20240426 09:55:24.731357 10083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 892) took 1066 ms. Trace:
I20240426 09:55:24.773084 10083 rpcz_store.cc:276] 0426 09:55:23.662483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.589248 (+926765us) service_pool.cc:225] Handling call
0426 09:55:24.728911 (+139663us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.773921 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:24.774441 10083 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:24.775367 10007 raft_consensus.cc:1226] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:24.774570 10014 raft_consensus.cc:1226] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:24.776368 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.776986 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.777401 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:24.778023 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.782985 11407 consensus_queue.cc:1046] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:24.783370 10015 raft_consensus.cc:1261] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:24.726606 10088 rpcz_store.cc:276] 0426 09:55:23.645931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.440916 (+794985us) service_pool.cc:225] Handling call
0426 09:55:24.676433 (+235517us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b5400083c54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:24.676571 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":503707520}
I20240426 09:55:24.684510 11454 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:24.708504 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:24.709439 11411 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:24.698403 11463 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.742264 10090 rpcz_store.cc:276] 0426 09:55:23.499768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:23.593008 (+ 93240us) service_pool.cc:225] Handling call
0426 09:55:24.678989 (+1085981us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b54001072d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:24.679110 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":630101888}
I20240426 09:55:24.742863 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:24.742584 10089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 880) took 1243 ms (client timeout 1623 ms). Trace:
I20240426 09:55:24.743655 10053 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:59762 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 121)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 891)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1058)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 120)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1059)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1062)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1065)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1064)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1066)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1063)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 123)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 906)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 886)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1057)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1067)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 106)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 107)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 108)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1038)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1040)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1041)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 114)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1042)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1045)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1046)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1047)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1048)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1051)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 117)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 868)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 869)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 870)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 876)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1054)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 883)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 885)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1055)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 889)
I20240426 09:55:24.707183 11404 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.751437 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 871) took 1293 ms. Trace:
I20240426 09:55:24.795388 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.761372 11582 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:24.648010 9904 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:24.724151 10055 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:59746 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 879)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 890)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 121)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 891)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1058)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 120)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1059)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1062)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1065)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1064)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1066)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1063)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 123)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 906)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 886)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1057)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1067)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 106)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 107)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 108)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1038)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1040)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1041)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 114)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1042)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1045)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1046)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1047)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1048)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1051)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 117)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 868)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 869)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 870)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 876)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1054)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 883)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 885)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1055)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 889)
I20240426 09:55:24.711266 11444 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:24.777931 10007 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
W20240426 09:55:24.777910 11589 log.cc:924] Time spent T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20240426 09:55:24.784407 9982 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:24.790477 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:24.835309 10088 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:24.835604 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.792351 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:24.794150 10618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51158 (request call id 141) took 1162 ms. Trace:
I20240426 09:55:24.797529 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 890) took 1142 ms. Trace:
I20240426 09:55:24.837025 10007 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:24.836732 10087 rpcz_store.cc:276] 0426 09:55:23.655327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.742618 (+1087291us) service_pool.cc:225] Handling call
0426 09:55:24.797476 (+ 54858us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.838567 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.816664 10086 raft_consensus.cc:2396] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 09:55:24.816728 10085 rpcz_store.cc:276] 0426 09:55:23.457656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.603707 (+1146051us) service_pool.cc:225] Handling call
0426 09:55:24.751389 (+147682us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.760370 11315 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:24.762554 10084 rpcz_store.cc:276] 0426 09:55:23.065823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.761031 (+1695208us) service_pool.cc:225] Handling call
0426 09:55:24.762050 (+ 1019us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:55:24.826903 10089 rpcz_store.cc:269] 0426 09:55:23.499457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:23.509852 (+ 10395us) service_pool.cc:225] Handling call
0426 09:55:24.730639 (+1220787us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x7b540037ee54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:24.742534 (+ 11895us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1307292544}
I20240426 09:55:24.849434 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:24.832055 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 184115us
I20240426 09:55:24.849879 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.850580 10090 raft_consensus.cc:2454] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:24.741914 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
W20240426 09:55:24.849447 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 65108.4us
I20240426 09:55:24.855664 11315 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:24.856174 11315 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.858207 11315 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:24.841462 10086 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 879) took 1350 ms. Trace:
I20240426 09:55:24.839721 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 896) took 1059 ms. Trace:
I20240426 09:55:24.754350 11461 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.845248 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:24.851984 9904 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:24.858659 10086 rpcz_store.cc:276] 0426 09:55:23.490640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.731330 (+1240690us) service_pool.cc:225] Handling call
0426 09:55:24.841406 (+110076us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.867336 10012 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:24.854394 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.867981 10089 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:24.868235 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:24.869201 10085 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:24.870083 9904 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:24.867733 10088 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:24.858713 10087 rpcz_store.cc:276] 0426 09:55:23.780674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.838359 (+1057685us) service_pool.cc:225] Handling call
0426 09:55:24.839652 (+ 1293us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.873101 10012 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:24.872857 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:24.875286 10088 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:24.876453 9904 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:24.882110 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:24.882678 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.883358 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.883968 10089 raft_consensus.cc:2454] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:24.883316 10087 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:24.818878 11403 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:24.834612 11320 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:24.885342 9906 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:24.892827 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.893982 9906 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:24.858408 11536 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:24.893952 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.758991 10081 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:24.818687 11408 consensus_queue.cc:1035] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20240426 09:55:24.791918 11575 consensus_queue.cc:1035] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.097s
I20240426 09:55:24.907223 10088 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:24.906625 9906 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:24.884739 11513 raft_consensus.cc:2741] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240426 09:55:24.892241 11461 raft_consensus.cc:2726] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:24.883791 11504 raft_consensus.cc:2726] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:24.910423 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
W20240426 09:55:24.912642 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 78842.3us
I20240426 09:55:24.892714 11474 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:24.855161 11422 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.859278 11350 consensus_queue.cc:1046] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20240426 09:55:24.903852 11328 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:24.916560 10088 raft_consensus.cc:2354] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3.
I20240426 09:55:24.899209 11536 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:24.903833 11521 consensus_queue.cc:1035] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20240426 09:55:24.889155 11490 consensus_queue.cc:1046] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.027s
I20240426 09:55:24.842025 10618 rpcz_store.cc:276] 0426 09:55:23.632126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:23.633367 (+ 1241us) service_pool.cc:225] Handling call
0426 09:55:24.793946 (+1160579us) spinlock_profiling.cc:243] Waited 672 ms on lock 0x7b5400450e54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:24.794079 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1546177792}
I20240426 09:55:24.928354 10010 raft_consensus.cc:1226] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:24.911849 10081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 891) took 1250 ms. Trace:
I20240426 09:55:24.914031 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.914399 11452 raft_consensus.cc:2781] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:24.942102 11404 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:24.925459 11536 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:24.925303 11422 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.934072 11545 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:24.965759 10055 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:24.947407 10084 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:24.899111 11403 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:24.947137 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:24.980567 10088 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:24.955655 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.983605 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.984215 10012 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:24.984973 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.985971 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.956001 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:24.987078 10015 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:24.987457 10088 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:24.988701 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:24.989822 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.990358 10088 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:24.956259 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.991200 10007 raft_consensus.cc:2379] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:24.991389 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.956516 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.991968 10088 raft_consensus.cc:2362] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3. Re-sending same reply.
I20240426 09:55:24.962889 9979 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:24.963076 10013 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:24.948428 9905 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:24.992576 10088 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 886) took 1352 ms. Trace:
I20240426 09:55:24.993522 11545 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:24.993460 10088 rpcz_store.cc:276] 0426 09:55:23.640611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.991174 (+1350563us) service_pool.cc:225] Handling call
0426 09:55:24.992527 (+ 1353us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:24.980566 11454 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:24.992580 10011 raft_consensus.cc:2379] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:24.994148 11545 raft_consensus.cc:686] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:24.994858 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.991894 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.995247 11545 consensus_queue.cc:237] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.993948 9979 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:24.996073 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:24.996764 10088 raft_consensus.cc:2362] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2. Re-sending same reply.
I20240426 09:55:24.995966 10007 raft_consensus.cc:2379] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:24.997072 11422 raft_consensus.cc:2741] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20240426 09:55:24.997057 9979 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:24.997612 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:24.998114 11422 raft_consensus.cc:2781] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:24.998435 9979 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:24.998466 11422 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:24.999109 11422 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:25.002934 11502 raft_consensus.cc:2781] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:25.003496 11502 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:25.003829 11502 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:25.002583 10015 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:25.002995 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:24.955806 11587 log.cc:924] Time spent T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
W20240426 09:55:25.017266 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 68887.6us
I20240426 09:55:24.948189 11504 raft_consensus.cc:3032] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Stepping down as leader of term 2
W20240426 09:55:24.955300 11592 log.cc:924] Time spent T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.163s user 0.001s sys 0.000s
I20240426 09:55:24.957055 10618 raft_consensus.cc:1261] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 09:55:24.957135 11463 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:25.027429 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.957233 11462 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:24.980408 11328 raft_consensus.cc:686] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:24.913242 11435 consensus_queue.cc:237] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:24.932653 10081 rpcz_store.cc:276] 0426 09:55:23.659044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.750695 (+1091651us) service_pool.cc:225] Handling call
0426 09:55:24.909583 (+158888us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.038053 10012 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:55:25.042846 11350 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:25.043418 11350 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.045423 11350 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:25.048316 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:24.979815 11452 raft_consensus.cc:686] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:25.023025 11490 raft_consensus.cc:2781] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election won for term 3
W20240426 09:55:25.062194 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 109571us
I20240426 09:55:24.954718 11582 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.022248 11504 raft_consensus.cc:729] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:24.959533 11524 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.071139 10583 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:25.071522 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:25.073308 10583 leader_election.cc:398] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3.
I20240426 09:55:25.075470 10583 leader_election.cc:401] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 09:55:25.075976 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:25.076653 11328 consensus_queue.cc:237] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.064220 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:25.077319 10583 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:25.077895 11394 raft_consensus.cc:2726] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3.
I20240426 09:55:25.078457 10583 leader_election.cc:398] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3.
I20240426 09:55:25.079708 10016 raft_consensus.cc:1226] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:25.073959 11597 raft_consensus.cc:2781] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:25.076648 10081 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:25.102033 10085 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:25.078483 11394 raft_consensus.cc:2726] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:55:25.103075 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:25.103284 9979 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:25.107940 10085 raft_consensus.cc:2379] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:25.108325 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 901) took 1138 ms. Trace:
I20240426 09:55:25.103380 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:25.084371 11490 raft_consensus.cc:686] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:25.111297 10009 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:25.108588 10085 rpcz_store.cc:276] 0426 09:55:23.969565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.102845 (+1133280us) service_pool.cc:225] Handling call
0426 09:55:25.108286 (+ 5441us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.085867 10084 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:25.112483 10055 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:25.131600 10055 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:25.132465 11343 raft_consensus.cc:2781] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:25.132848 11343 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:25.092986 11320 raft_consensus.cc:2726] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:25.133210 11343 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:25.133658 11411 raft_consensus.cc:2726] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:25.117738 11452 consensus_queue.cc:237] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.136286 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:25.137102 10090 raft_consensus.cc:2379] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:25.137492 10090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 882) took 1633 ms. Trace:
I20240426 09:55:25.119570 11520 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:25.137785 10090 rpcz_store.cc:276] 0426 09:55:23.503639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.907267 (+1403628us) service_pool.cc:225] Handling call
0426 09:55:25.136104 (+228837us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x7b58000b0498. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:55:25.137436 (+ 1332us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4642048}
I20240426 09:55:25.120843 11513 raft_consensus.cc:3032] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:25.121915 11582 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:25.122433 11524 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:25.074096 11533 consensus_queue.cc:1035] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20240426 09:55:25.121802 11474 raft_consensus.cc:2726] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:25.124461 11490 consensus_queue.cc:237] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:25.139096 10055 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:25.139405 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:25.140906 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:25.142211 10009 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
W20240426 09:55:25.159340 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 53855.1us
I20240426 09:55:25.077592 11504 consensus_queue.cc:260] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.134550 11336 raft_consensus.cc:2726] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 09:55:25.155521 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 52336.2us
I20240426 09:55:25.096792 11597 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:25.163563 10087 raft_consensus.cc:2379] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:25.176254 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 895) took 1470 ms. Trace:
I20240426 09:55:25.176578 10087 rpcz_store.cc:276] 0426 09:55:23.705917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.139178 (+1433261us) service_pool.cc:225] Handling call
0426 09:55:25.176195 (+ 37017us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.164005 10090 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:25.177402 10090 raft_consensus.cc:595] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:25.177750 10090 raft_consensus.cc:603] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 09:55:25.122825 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:25.116480 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:25.178819 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 900) took 1211 ms. Trace:
I20240426 09:55:25.143838 11513 raft_consensus.cc:729] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:25.179013 10055 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:25.179145 10085 rpcz_store.cc:276] 0426 09:55:23.967987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.116295 (+1148308us) service_pool.cc:225] Handling call
0426 09:55:25.178749 (+ 62454us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.184036 10016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1254) took 2245 ms. Trace:
I20240426 09:55:25.186141 9906 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:25.187197 10087 raft_consensus.cc:1226] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:25.159104 11519 raft_consensus.cc:2781] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:25.187896 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 106) took 2284 ms. Trace:
I20240426 09:55:25.198846 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:25.187240 10016 rpcz_store.cc:276] 0426 09:55:22.938577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.478612 (+1540035us) service_pool.cc:225] Handling call
0426 09:55:24.713991 (+235379us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b580004ec98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:55:24.714140 (+ 149us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.083810 (+369670us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.183815 (+100005us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x7b540016bf54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:25.183974 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":848688384}
I20240426 09:55:25.182657 11404 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20240426 09:55:25.193572 11457 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:25.194304 11473 raft_consensus.cc:2781] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:25.195880 10083 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:25.199647 10085 raft_consensus.cc:2354] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
I20240426 09:55:25.199297 10087 rpcz_store.cc:276] 0426 09:55:22.903073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.185188 (+2282115us) service_pool.cc:225] Handling call
0426 09:55:25.185458 (+ 270us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.187695 (+ 2237us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.187850 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.216742 10087 raft_consensus.cc:1226] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:25.166291 11454 raft_consensus.cc:2781] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:25.206760 11457 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:55:25.209712 11595 log.cc:924] Time spent T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.415s user 0.005s sys 0.000s
I20240426 09:55:25.211170 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:25.212106 10016 raft_consensus.cc:1261] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:25.183001 11597 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:25.218004 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1038) took 2299 ms. Trace:
I20240426 09:55:25.219164 9979 leader_election.cc:398] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
I20240426 09:55:25.219602 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 108) took 2316 ms. Trace:
I20240426 09:55:25.236091 9979 leader_election.cc:401] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 09:55:25.236105 10085 rpcz_store.cc:276] 0426 09:55:22.918445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.216455 (+2298010us) service_pool.cc:225] Handling call
0426 09:55:25.217598 (+ 1143us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.217744 (+ 146us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:25.217767 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:25.217804 (+ 37us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:25.217831 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.217876 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:25.217960 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.236234 10087 rpcz_store.cc:276] 0426 09:55:22.903554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.216337 (+2312783us) service_pool.cc:225] Handling call
0426 09:55:25.216576 (+ 239us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.217119 (+ 543us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.219555 (+ 2436us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.236806 10013 raft_consensus.cc:1226] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:25.221601 11473 raft_consensus.cc:686] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:25.235580 11457 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:25.219787 10083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1059) took 1237 ms. Trace:
I20240426 09:55:25.241014 11517 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:25.214079 11519 raft_consensus.cc:686] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:25.243571 11454 raft_consensus.cc:686] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:25.284929 10081 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:25.267807 10083 rpcz_store.cc:276] 0426 09:55:23.982417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.773704 (+791287us) service_pool.cc:225] Handling call
0426 09:55:25.217291 (+443587us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.288368 10083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1042) took 2327 ms. Trace:
I20240426 09:55:25.288702 10083 rpcz_store.cc:276] 0426 09:55:22.969312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.287600 (+2318288us) service_pool.cc:225] Handling call
0426 09:55:25.287888 (+ 288us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.288056 (+ 168us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:25.288087 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:25.288125 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:25.288159 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.288215 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:25.288317 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.292042 11519 consensus_queue.cc:237] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.286628 10081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 114) took 2328 ms. Trace:
I20240426 09:55:25.295571 10081 rpcz_store.cc:276] 0426 09:55:22.958623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.285781 (+2327158us) service_pool.cc:225] Handling call
0426 09:55:25.286125 (+ 344us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.286310 (+ 185us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:25.286337 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:25.286381 (+ 44us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:25.286416 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.286470 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:25.286574 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.299508 10081 raft_consensus.cc:1226] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 09:55:25.298233 11457 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.301266 10084 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:25.301477 11504 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:25.301607 10087 raft_consensus.cc:1226] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:25.250345 11411 consensus_queue.cc:1046] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.053s
I20240426 09:55:25.302920 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1041) took 2367 ms. Trace:
I20240426 09:55:25.300954 10081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1046) took 2252 ms. Trace:
I20240426 09:55:25.303651 11336 raft_consensus.cc:3032] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:25.304044 11336 raft_consensus.cc:729] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:25.304231 11457 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:25.304970 11457 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:25.305393 11457 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.309121 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:25.303268 10087 rpcz_store.cc:276] 0426 09:55:22.935681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.239530 (+2303849us) service_pool.cc:225] Handling call
0426 09:55:25.239855 (+ 325us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.302040 (+ 62185us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.302718 (+ 678us) spinlock_profiling.cc:243] Waited 61.6 ms on lock 0x7b540037e954. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:25.302876 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":141666560}
I20240426 09:55:25.310626 10087 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:25.311331 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1047) took 2256 ms. Trace:
I20240426 09:55:25.311631 10087 rpcz_store.cc:276] 0426 09:55:23.054648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.310054 (+2255406us) service_pool.cc:225] Handling call
0426 09:55:25.310309 (+ 255us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.310997 (+ 688us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.311280 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.320909 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1256) took 2308 ms. Trace:
I20240426 09:55:25.321346 10010 rpcz_store.cc:276] 0426 09:55:23.012948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.495212 (+1482264us) service_pool.cc:225] Handling call
0426 09:55:24.730081 (+234869us) spinlock_profiling.cc:243] Waited 7.12 ms on lock 0x7b580004ec98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:55:24.742148 (+ 12067us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:24.930193 (+188045us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.320673 (+390480us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b5400056ed4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:25.320850 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":438112512}
I20240426 09:55:25.244513 11473 consensus_queue.cc:237] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.333448 10015 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:25.335636 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1048) took 2090 ms. Trace:
I20240426 09:55:25.345271 10087 rpcz_store.cc:276] 0426 09:55:23.245203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.334792 (+2089589us) service_pool.cc:225] Handling call
0426 09:55:25.335129 (+ 337us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.335307 (+ 178us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:25.335333 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:25.335373 (+ 40us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:25.335400 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.335462 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:25.335583 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.346477 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:25.347987 10087 raft_consensus.cc:1261] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:25.348510 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1051) took 1954 ms. Trace:
I20240426 09:55:25.299410 11403 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:25.276021 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:25.298290 11536 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.376315 10011 raft_consensus.cc:1226] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:25.377296 11536 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:25.352933 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:25.250905 11402 consensus_queue.cc:1046] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20240426 09:55:25.352800 10055 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:25.378904 11462 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:25.379369 11462 raft_consensus.cc:686] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:25.380183 11462 consensus_queue.cc:237] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.381875 10015 raft_consensus.cc:1261] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:25.383108 11454 consensus_queue.cc:237] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:25.383590 11315 consensus_queue.cc:1035] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:25.242887 11513 consensus_queue.cc:260] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.250509 11609 consensus_queue.cc:1035] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.111s
I20240426 09:55:25.368731 10084 raft_consensus.cc:1261] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 09:55:25.394361 10084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 117) took 1994 ms. Trace:
I20240426 09:55:25.395537 10010 raft_consensus.cc:1261] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:25.394750 10084 rpcz_store.cc:276] 0426 09:55:23.400122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.355704 (+1955582us) service_pool.cc:225] Handling call
0426 09:55:25.368341 (+ 12637us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.394056 (+ 25715us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.394302 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.370615 10082 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
I20240426 09:55:25.372092 11343 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.373019 11336 consensus_queue.cc:260] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.363593 11422 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.363175 11404 consensus_queue.cc:1046] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.012s
W20240426 09:55:25.399252 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 80810.3us
I20240426 09:55:25.356504 11457 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:25.348826 10087 rpcz_store.cc:276] 0426 09:55:23.394060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.347395 (+1953335us) service_pool.cc:225] Handling call
0426 09:55:25.347668 (+ 273us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.348288 (+ 620us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.348460 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.403244 10087 raft_consensus.cc:1226] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240426 09:55:25.403959 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 869) took 1946 ms. Trace:
I20240426 09:55:25.404312 10087 rpcz_store.cc:276] 0426 09:55:23.457196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.402200 (+1945004us) service_pool.cc:225] Handling call
0426 09:55:25.402509 (+ 309us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:25.403710 (+ 1201us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.403907 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.303548 10081 rpcz_store.cc:276] 0426 09:55:23.048826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.298507 (+2249681us) service_pool.cc:225] Handling call
0426 09:55:25.299267 (+ 760us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:25.300688 (+ 1421us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.300901 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.411633 10085 raft_consensus.cc:1226] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:25.415052 10087 raft_consensus.cc:1261] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:25.415498 10009 raft_consensus.cc:1226] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:25.415870 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 870) took 1958 ms. Trace:
I20240426 09:55:25.416297 10087 rpcz_store.cc:276] 0426 09:55:23.457418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.405402 (+1947984us) service_pool.cc:225] Handling call
0426 09:55:25.405658 (+ 256us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.415556 (+ 9898us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.415760 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.419075 9979 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
W20240426 09:55:25.397881 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 154225us
I20240426 09:55:25.420434 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:25.423244 9979 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:25.417076 11490 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:25.400318 11411 consensus_queue.cc:1035] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20240426 09:55:25.430464 11502 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.432202 11410 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:25.432461 11502 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:25.418154 11336 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:25.438922 10087 raft_consensus.cc:1261] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:25.439916 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1054) took 1945 ms. Trace:
I20240426 09:55:25.439993 9979 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:25.440265 10087 rpcz_store.cc:276] 0426 09:55:23.494916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.438315 (+1943399us) service_pool.cc:225] Handling call
0426 09:55:25.438583 (+ 268us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.439405 (+ 822us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.439867 (+ 462us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.441533 10087 raft_consensus.cc:1226] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:25.426577 11617 raft_consensus.cc:3032] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:25.443888 10586 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:25.427131 11545 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:25.451023 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 883) took 1942 ms. Trace:
I20240426 09:55:25.451424 10087 rpcz_store.cc:276] 0426 09:55:23.590938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.441111 (+1850173us) service_pool.cc:225] Handling call
0426 09:55:25.441380 (+ 269us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.450739 (+ 9359us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.450967 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.459867 11343 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:25.436548 10084 raft_consensus.cc:1226] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240426 09:55:25.460789 10084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 868) took 2003 ms. Trace:
I20240426 09:55:25.461194 11618 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:25.461144 10084 rpcz_store.cc:276] 0426 09:55:23.456960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.396356 (+1939396us) service_pool.cc:225] Handling call
0426 09:55:25.418466 (+ 22110us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:25.460521 (+ 42055us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.460734 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.463007 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:25.445325 11502 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:25.464435 10090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 107) took 2561 ms. Trace:
I20240426 09:55:25.465047 10090 rpcz_store.cc:276] 0426 09:55:22.903329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.183960 (+2280631us) service_pool.cc:225] Handling call
0426 09:55:25.184293 (+ 333us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:25.184617 (+ 324us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:25.184645 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:25.190890 (+ 6245us) log.cc:842] Serialized 26 byte log entry
0426 09:55:25.398195 (+207305us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:25.398229 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.398290 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:25.464227 (+ 65937us) raft_consensus.cc:1671] finished
0426 09:55:25.464249 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:25.464383 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":98,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":10348,"thread_start_us":162433,"threads_started":1,"wal-append.queue_time_us":221195}
I20240426 09:55:25.467103 10088 raft_consensus.cc:2396] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 09:55:25.441869 11533 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20240426 09:55:25.442219 11623 consensus_queue.cc:1035] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
W20240426 09:55:25.446831 11444 consensus_meta.cc:287] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.603s user 0.006s sys 0.003s
I20240426 09:55:25.443149 11410 raft_consensus.cc:686] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:25.446557 11617 raft_consensus.cc:729] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:25.471607 10612 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:25.472034 11410 consensus_queue.cc:237] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.471525 11617 consensus_queue.cc:260] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:25.475056 11517 raft_consensus.cc:2726] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3.
I20240426 09:55:25.515039 10087 raft_consensus.cc:1226] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240426 09:55:25.475396 11617 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:25.518357 10090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 889) took 1864 ms. Trace:
I20240426 09:55:25.518656 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 885) took 1882 ms. Trace:
I20240426 09:55:25.519891 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:25.521183 10013 raft_consensus.cc:2354] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4.
I20240426 09:55:25.520443 10090 rpcz_store.cc:276] 0426 09:55:23.653825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.466493 (+1812668us) service_pool.cc:225] Handling call
0426 09:55:25.517883 (+ 51390us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.518063 (+ 180us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:25.518085 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:25.518122 (+ 37us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:25.518150 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.518198 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:25.518306 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:55:25.522409 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 44368.1us
I20240426 09:55:25.520973 10087 rpcz_store.cc:276] 0426 09:55:23.636393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.513732 (+1877339us) service_pool.cc:225] Handling call
0426 09:55:25.514372 (+ 640us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:25.518333 (+ 3961us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.518598 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.527004 10090 raft_consensus.cc:1261] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 09:55:25.526965 10055 leader_election.cc:398] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4.
I20240426 09:55:25.527637 10055 leader_election.cc:401] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20240426 09:55:25.528539 11230 consensus_queue.cc:1046] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 09:55:25.529759 11422 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:25.516239 11502 raft_consensus.cc:686] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:25.530963 11502 consensus_queue.cc:237] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.534247 11607 consensus_queue.cc:1035] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240426 09:55:25.524925 11613 log.cc:924] Time spent T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.108s user 0.000s sys 0.001s
I20240426 09:55:25.517616 11545 raft_consensus.cc:686] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:25.441291 11490 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.547180 10016 raft_consensus.cc:1261] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:25.565742 10090 raft_consensus.cc:1261] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:25.541535 11343 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:55:25.567114 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 122963us
I20240426 09:55:25.566074 11404 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:25.548450 11422 consensus_queue.cc:1035] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20240426 09:55:25.588573 10090 raft_consensus.cc:1226] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:25.592736 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:25.594389 10089 raft_consensus.cc:1261] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 09:55:25.597123 10089 raft_consensus.cc:1261] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:25.572350 10084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1055) took 1919 ms. Trace:
W20240426 09:55:25.598299 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 712047us
I20240426 09:55:25.601150 9904 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:25.603739 10090 raft_consensus.cc:1226] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:25.569021 11444 consensus_queue.cc:1035] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.024s
I20240426 09:55:25.589270 11504 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:25.605156 9904 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:25.608230 11600 raft_consensus.cc:2781] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:25.590405 11404 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:25.642529 11597 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:25.646306 11585 raft_consensus.cc:3032] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:25.646687 11585 raft_consensus.cc:729] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
W20240426 09:55:25.655493 11626 log.cc:924] Time spent T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
I20240426 09:55:25.651423 10618 raft_consensus.cc:1226] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:25.630156 11328 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:25.666157 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:25.666878 10008 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:25.604768 11545 consensus_queue.cc:237] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.599395 10084 rpcz_store.cc:276] 0426 09:55:23.652290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.462603 (+1810313us) service_pool.cc:225] Handling call
0426 09:55:25.518597 (+ 55994us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.538265 (+ 19668us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:25.541547 (+ 3282us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:25.544669 (+ 3122us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:25.551029 (+ 6360us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.564864 (+ 13835us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:25.571214 (+ 6350us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.659190 11462 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:25.671932 11462 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:25.625629 11504 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.660923 11597 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:25.662178 11490 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:25.662482 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:55:25.665169 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 60088.1us
I20240426 09:55:25.684250 9904 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:25.685432 9904 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:25.686164 11474 raft_consensus.cc:2741] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20240426 09:55:25.686751 11474 raft_consensus.cc:2781] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:25.687106 11474 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:25.676142 10013 raft_consensus.cc:1261] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:25.676802 9906 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:25.689311 11452 raft_consensus.cc:2781] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:25.689662 11452 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:25.689965 11452 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:25.694906 11435 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:25.695353 11435 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.697201 11435 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:25.697914 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:25.644270 11600 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:25.673905 11462 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:25.675310 11504 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:25.708278 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:25.708559 11609 consensus_queue.cc:1035] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:25.702211 11474 raft_consensus.cc:3037] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:25.701750 9904 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:25.699352 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:25.675335 11536 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:25.719419 9904 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:25.678905 11404 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:25.723269 9904 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:25.724565 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:25.726074 10007 raft_consensus.cc:2379] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:25.724658 11504 raft_consensus.cc:2781] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:25.722983 11609 raft_consensus.cc:2726] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 09:55:25.730249 11629 log.cc:924] Time spent T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.155s user 0.003s sys 0.000s
I20240426 09:55:25.735219 11461 raft_consensus.cc:2726] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:25.736025 11504 raft_consensus.cc:686] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:25.736945 11504 consensus_queue.cc:237] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.725265 11536 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.740545 11536 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:25.742583 10015 raft_consensus.cc:1261] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:25.707700 11600 raft_consensus.cc:3037] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:25.745210 9904 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:25.747810 11381 raft_consensus.cc:3032] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:25.749519 11381 raft_consensus.cc:729] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:25.750028 11609 raft_consensus.cc:2726] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:25.753074 11343 raft_consensus.cc:2726] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4.
I20240426 09:55:25.752823 11381 consensus_queue.cc:260] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.751096 9904 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:25.723482 11585 consensus_queue.cc:260] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.759097 9904 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:25.764570 9904 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:25.764721 11253 raft_consensus.cc:2781] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:25.793506 9904 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:25.762925 11578 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:25.765444 10013 raft_consensus.cc:1261] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:25.796721 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:25.799115 10086 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1063) took 1288 ms. Trace:
I20240426 09:55:25.799438 10086 rpcz_store.cc:276] 0426 09:55:24.511116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.796485 (+1285369us) service_pool.cc:225] Handling call
0426 09:55:25.799056 (+ 2571us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:25.784489 11381 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:25.764045 11490 consensus_queue.cc:1035] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.045s
I20240426 09:55:25.810380 11452 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.812500 11452 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:25.802059 10083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1045) took 2787 ms. Trace:
I20240426 09:55:25.815264 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:25.814422 10083 rpcz_store.cc:276] 0426 09:55:23.015144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.289542 (+2274398us) service_pool.cc:225] Handling call
0426 09:55:25.289848 (+ 306us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:25.290184 (+ 336us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:25.290214 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:25.290646 (+ 432us) log.cc:842] Serialized 26 byte log entry
0426 09:55:25.537178 (+246532us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:25.538260 (+ 1082us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:25.543697 (+ 5437us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:25.729764 (+186067us) spinlock_profiling.cc:243] Waited 47.1 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:55:25.792996 (+ 63232us) raft_consensus.cc:1671] finished
0426 09:55:25.794486 (+ 1490us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:25.801988 (+ 7502us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13311,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"spinlock_wait_cycles":108301568,"thread_start_us":178110,"threads_started":1,"wal-append.queue_time_us":257587}
I20240426 09:55:25.815819 10012 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:25.801565 11578 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:25.816330 11578 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:25.788655 11585 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:25.830682 11490 raft_consensus.cc:2781] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:55:25.831147 11490 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:25.831485 11490 raft_consensus.cc:3037] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:25.818033 11328 raft_consensus.cc:2726] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:25.800894 11435 raft_consensus.cc:2781] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:25.822814 10012 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:25.762573 11474 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.837287 9906 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:25.860831 10088 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1057) took 2081 ms. Trace:
I20240426 09:55:25.861243 10088 rpcz_store.cc:276] 0426 09:55:23.779658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.997379 (+1217721us) service_pool.cc:225] Handling call
0426 09:55:25.860626 (+863247us) spinlock_profiling.cc:243] Waited 469 ms on lock 0x7b54001815d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:25.860770 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1077664512}
I20240426 09:55:25.846998 11435 raft_consensus.cc:686] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:25.839030 11578 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.853485 11444 raft_consensus.cc:2781] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:25.835574 11461 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20240426 09:55:25.892336 10613 raft_consensus.cc:1261] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 09:55:25.867341 11328 consensus_queue.cc:1046] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20240426 09:55:25.875351 11444 raft_consensus.cc:686] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:25.885517 10081 raft_consensus.cc:1261] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
W20240426 09:55:25.867866 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 74392.2us
I20240426 09:55:25.902949 10081 pending_rounds.cc:76] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee: Aborting all ops after (but not including) 1
I20240426 09:55:25.922632 11646 consensus_queue.cc:1035] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 09:55:25.894956 11600 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:25.908402 10081 pending_rounds.cc:97] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240426 09:55:25.911700 9904 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:25.930414 11474 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:25.913959 11253 raft_consensus.cc:686] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:25.935039 9941 raft_consensus.cc:1226] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:25.908627 11435 consensus_queue.cc:237] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:25.943231 11253 consensus_queue.cc:237] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.943634 9904 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:25.929178 10081 raft_consensus.cc:2866] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:25.927141 11578 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:25.954818 10615 raft_consensus.cc:1261] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240426 09:55:25.955456 11578 raft_consensus.cc:2726] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:25.931368 11444 consensus_queue.cc:237] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.956807 9904 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:25.958240 11520 consensus_queue.cc:1035] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240426 09:55:25.952433 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 58805.5us
W20240426 09:55:25.961764 11643 log.cc:924] Time spent T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240426 09:55:25.968252 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:25.954841 11435 raft_consensus.cc:2781] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:25.972949 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:25.973523 9939 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:25.976536 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:25.977097 10014 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:25.979388 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:25.963244 11578 raft_consensus.cc:2726] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:25.986601 10014 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:25.958621 11490 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:25.989876 11644 consensus_queue.cc:1035] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240426 09:55:25.999415 10611 raft_consensus.cc:1226] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:26.001803 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.002398 9932 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:25.975325 11435 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:26.021188 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:25.992249 11600 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:25.989696 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.008069 11435 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:26.077687 11578 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:26.081216 11650 log.cc:924] Time spent T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20240426 09:55:26.077836 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:55:26.105088 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 127764us
I20240426 09:55:26.107053 9906 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:55:26.108472 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 65196.9us
W20240426 09:55:26.109099 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 85514.8us
I20240426 09:55:26.034257 11408 consensus_queue.cc:1046] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20240426 09:55:26.080598 9934 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:26.106392 9932 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:26.093539 11578 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:26.122094 11578 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:26.129824 11230 raft_consensus.cc:3032] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:26.130255 11230 raft_consensus.cc:729] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:26.132462 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:26.133689 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.134389 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.135056 9941 raft_consensus.cc:2454] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:26.135284 11402 consensus_queue.cc:1046] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:26.136358 11402 raft_consensus.cc:3032] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:26.136679 11402 raft_consensus.cc:729] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:26.134584 11230 consensus_queue.cc:260] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:26.137408 11402 consensus_queue.cc:260] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:26.138640 11402 raft_consensus.cc:3037] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:26.142045 11230 raft_consensus.cc:3037] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:26.144619 10087 raft_consensus.cc:1226] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: []
W20240426 09:55:26.144536 10082 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 120) took 1792 ms (client timeout 1749 ms). Trace:
I20240426 09:55:26.145206 11474 raft_consensus.cc:3032] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:26.145609 11474 raft_consensus.cc:729] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:26.145891 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1075) took 1194 ms. Trace:
I20240426 09:55:26.146222 10087 rpcz_store.cc:276] 0426 09:55:24.951980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.535431 (+583451us) service_pool.cc:225] Handling call
0426 09:55:26.144446 (+609015us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:26.145039 (+ 593us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:26.145729 (+ 690us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b5400181ad0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c45b99 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:26.145843 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1397120896}
I20240426 09:55:26.132376 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1040) took 3207 ms. Trace:
I20240426 09:55:26.147420 10085 rpcz_store.cc:276] 0426 09:55:22.925258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.237823 (+2312565us) service_pool.cc:225] Handling call
0426 09:55:25.241060 (+ 3237us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.416325 (+175265us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:26.132156 (+715831us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b540024f754. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:26.132321 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":391867264}
W20240426 09:55:26.145220 10082 rpcz_store.cc:269] 0426 09:55:24.352351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.763189 (+410838us) service_pool.cc:225] Handling call
0426 09:55:26.144332 (+1381143us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b5400181ad4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:26.144451 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":997397760}
I20240426 09:55:26.150206 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.150971 11490 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:26.151369 9941 raft_consensus.cc:2354] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4.
W20240426 09:55:26.153020 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 46046us
I20240426 09:55:26.151765 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:26.161370 11457 raft_consensus.cc:3032] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:26.161748 11457 raft_consensus.cc:729] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:26.162510 11457 consensus_queue.cc:260] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:26.163530 11457 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:26.168535 11533 consensus_queue.cc:1035] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:26.168787 9979 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:26.169251 11585 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:26.169813 11533 raft_consensus.cc:2726] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:26.070834 11653 consensus_queue.cc:1046] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20240426 09:55:26.171324 9979 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:26.173450 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:26.172500 9939 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:26.168962 11408 consensus_queue.cc:1035] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20240426 09:55:26.176368 10015 raft_consensus.cc:1226] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:26.178040 11545 raft_consensus.cc:2726] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:26.178184 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:26.181054 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:26.181227 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.182077 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:26.182752 9939 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:26.184252 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.186275 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:26.188942 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:26.189184 10015 raft_consensus.cc:1226] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:26.189914 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.192247 9941 raft_consensus.cc:2454] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:26.190045 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:26.180387 11253 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:26.225332 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.224879 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:26.229822 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.186010 11454 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:26.234422 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:26.235443 9941 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:26.240236 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:26.241274 9939 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:26.195657 11404 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:26.195452 11520 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:26.146293 11474 consensus_queue.cc:260] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:26.267766 9941 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:26.232352 11336 consensus_queue.cc:1046] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20240426 09:55:26.286803 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.257259 11253 raft_consensus.cc:686] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:26.254487 11416 consensus_queue.cc:1035] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
W20240426 09:55:26.266881 11519 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.969s user 0.010s sys 0.014s
I20240426 09:55:26.359318 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:26.241477 11454 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:26.292760 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:26.361022 9890 thread.cc:636] client-negotiator [worker] (thread pool) Time spent creating pthread: real 1.056s user 0.065s sys 0.054s
I20240426 09:55:26.406733 9939 raft_consensus.cc:2379] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:26.407606 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:26.291195 11404 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:26.440341 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.441473 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:55:26.403057 11519 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.103s user 0.010s sys 0.014s
I20240426 09:55:26.400053 11577 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:26.449896 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 267910us
I20240426 09:55:26.399871 11663 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:26.293277 11520 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:26.408794 9890 thread.cc:602] client-negotiator [worker] (thread pool) Time spent starting thread: real 1.104s user 0.065s sys 0.054s
W20240426 09:55:26.603950 11671 log.cc:924] Time spent T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
W20240426 09:55:26.465288 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 240590us
I20240426 09:55:26.473237 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.609432 9941 raft_consensus.cc:2454] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:26.610648 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:26.614851 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.615502 9939 raft_consensus.cc:2454] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
W20240426 09:55:26.572289 11675 log.cc:924] Time spent T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20240426 09:55:26.543717 11672 log.cc:924] Time spent T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.079s user 0.000s sys 0.001s
W20240426 09:55:26.524596 11664 log.cc:924] Time spent T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.164s user 0.000s sys 0.000s
W20240426 09:55:26.574082 11669 log.cc:924] Time spent T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.283s user 0.001s sys 0.000s
I20240426 09:55:26.584981 11404 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:26.521135 10616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51158 (request call id 150) took 1074 ms. Trace:
I20240426 09:55:26.602128 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:26.631206 10055 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
W20240426 09:55:26.581138 10612 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51158 (request call id 147) took 1845 ms (client timeout 2008 ms). Trace:
I20240426 09:55:26.626273 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:26.633680 11404 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:26.636516 10055 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:26.633270 11681 raft_consensus.cc:3032] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:26.659967 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:26.660569 10086 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:26.663237 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:26.663875 10090 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:26.672854 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:26.525580 11435 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:26.673424 10085 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:26.681737 9982 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:26.682848 9982 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:26.687585 9982 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:26.689024 9982 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:26.613755 11577 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:26.693706 10090 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:26.693948 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.694759 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:26.695528 10090 raft_consensus.cc:2379] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:26.696314 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:26.696501 10056 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:26.696872 10090 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:26.697715 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:26.695434 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.516430 11253 consensus_queue.cc:237] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:26.703791 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.704133 10013 raft_consensus.cc:1261] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20240426 09:55:26.712105 10013 pending_rounds.cc:76] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting all ops after (but not including) 1
I20240426 09:55:26.712343 10013 pending_rounds.cc:97] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240426 09:55:26.707460 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:26.713804 10013 raft_consensus.cc:2866] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:26.710408 10085 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
I20240426 09:55:26.614527 11663 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:26.636983 11680 raft_consensus.cc:2726] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:26.749441 9979 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:26.654036 11679 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:26.750154 11457 consensus_meta.cc:287] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.586s user 0.013s sys 0.000s
I20240426 09:55:26.760529 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:26.693240 11422 raft_consensus.cc:2726] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:26.767266 10086 raft_consensus.cc:2454] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:26.700040 11644 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:26.702232 11381 raft_consensus.cc:2781] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:26.673298 11681 raft_consensus.cc:729] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
W20240426 09:55:26.765195 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 129299us
I20240426 09:55:26.756310 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:26.712050 11545 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:26.779151 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:26.783951 10086 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 5
I20240426 09:55:26.628528 10616 rpcz_store.cc:276] 0426 09:55:25.443567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.444507 (+ 940us) service_pool.cc:225] Handling call
0426 09:55:25.549105 (+104598us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.568844 (+ 19739us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 09:55:25.625250 (+ 56406us) log.cc:842] Serialized 14 byte log entry
0426 09:55:26.148780 (+523530us) spinlock_profiling.cc:243] Waited 5.85 ms on lock 0x7b1800286a20. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fcfdb75 00007f9ca8b37789 00007f9c9fd1b0eb 00007f9c9fd194ef 00007f9c9fd1b652 00007f9ca28c743b 00007f9ca28ccee3 00007f9ca28cd3fb 00007f9ca2c50a1a
0426 09:55:26.402182 (+253402us) log.cc:842] Serialized 14 byte log entry
0426 09:55:26.446752 (+ 44570us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:26.452536 (+ 5784us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:55:26.474415 (+ 21879us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:26.496856 (+ 22441us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:26.517432 (+ 20576us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64,"prepare.queue_time_us":32347,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":4752,"raft.queue_time_us":42457,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":84851,"spinlock_wait_cycles":79969152,"thread_start_us":119194,"threads_started":1,"wal-append.queue_time_us":887574}
I20240426 09:55:26.676007 11404 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:26.789600 11681 consensus_queue.cc:260] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:26.786028 11545 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:26.703236 11424 consensus_queue.cc:1046] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
I20240426 09:55:26.794428 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.798804 10086 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:26.799845 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:26.822252 10085 raft_consensus.cc:2362] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5. Re-sending same reply.
I20240426 09:55:26.808687 10618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51158 (request call id 149) took 1517 ms. Trace:
I20240426 09:55:26.780839 11454 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:26.813244 11463 raft_consensus.cc:2726] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:26.783736 11381 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:26.786334 9934 raft_consensus.cc:2379] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
W20240426 09:55:26.636729 10612 rpcz_store.cc:269] 0426 09:55:24.728233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.728642 (+ 409us) service_pool.cc:225] Handling call
0426 09:55:26.522953 (+1794311us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b540049fcd4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:26.573536 (+ 50583us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":412495744}
I20240426 09:55:26.703248 11427 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:26.680790 11686 raft_consensus.cc:2726] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:26.788949 11644 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:26.862439 11644 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:26.825631 11681 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:26.830662 11381 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:26.771502 11657 consensus_queue.cc:1035] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20240426 09:55:26.897284 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:26.780201 9939 raft_consensus.cc:2396] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 09:55:26.777086 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:26.822510 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:26.784938 11543 raft_consensus.cc:2781] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:26.824075 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:26.917184 10085 raft_consensus.cc:2396] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240426 09:55:26.825522 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.918208 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.812526 11435 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:26.814271 11577 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:26.820086 11545 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:26.839007 11454 raft_consensus.cc:3032] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Stepping down as leader of term 3
W20240426 09:55:26.839056 9932 consensus_meta.cc:287] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.668s user 0.009s sys 0.001s
I20240426 09:55:26.908342 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:26.926327 10086 raft_consensus.cc:3037] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:26.918503 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.931072 9939 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:26.903455 11680 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:26.842226 9904 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
W20240426 09:55:26.924055 10011 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.746s user 0.023s sys 0.077s
I20240426 09:55:26.875463 11520 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:26.925119 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:26.931870 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.919499 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.778437 11679 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:26.934371 10086 raft_consensus.cc:2454] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:26.907640 11427 raft_consensus.cc:686] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
W20240426 09:55:26.952585 10011 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.774s user 0.023s sys 0.077s
I20240426 09:55:26.957058 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:26.957871 10086 raft_consensus.cc:2379] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:26.823467 10618 rpcz_store.cc:276] 0426 09:55:25.290850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.291056 (+ 206us) service_pool.cc:225] Handling call
0426 09:55:25.291364 (+ 308us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.671127 (+379763us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:26.808453 (+1137326us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b5400601654. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:26.808628 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":818213760}
I20240426 09:55:26.962381 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:26.964031 9936 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:26.964319 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:26.964974 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:55:26.966153 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 123999us
I20240426 09:55:26.966735 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:26.966902 9904 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:26.968767 9904 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:26.969221 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.963029 9935 raft_consensus.cc:2362] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3. Re-sending same reply.
I20240426 09:55:26.971383 9939 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:26.971951 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:26.972149 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:26.960799 10081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 876) took 3502 ms. Trace:
I20240426 09:55:26.972862 9939 raft_consensus.cc:2379] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 2: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:26.974615 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:26.924463 11454 raft_consensus.cc:729] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:26.925623 11543 raft_consensus.cc:686] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:26.977552 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:26.978235 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:26.979164 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 282784us
I20240426 09:55:26.979636 10090 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
W20240426 09:55:26.942293 11667 log.cc:924] Time spent T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.424s user 0.001s sys 0.003s
I20240426 09:55:26.922133 10083 raft_consensus.cc:2362] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4. Re-sending same reply.
I20240426 09:55:26.983698 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:26.932327 9932 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:26.985750 9939 raft_consensus.cc:1261] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:26.986421 9939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1048) took 1018 ms. Trace:
I20240426 09:55:26.986807 9939 rpcz_store.cc:276] 0426 09:55:25.968379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:26.985218 (+1016839us) service_pool.cc:225] Handling call
0426 09:55:26.985469 (+ 251us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:26.986217 (+ 748us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:26.986377 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:26.967339 10086 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:26.992509 9904 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:26.990226 10090 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:26.992600 9935 raft_consensus.cc:1226] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:26.993625 9904 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:26.994140 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1049) took 1025 ms. Trace:
I20240426 09:55:26.994552 9935 rpcz_store.cc:276] 0426 09:55:25.968556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:26.992215 (+1023659us) service_pool.cc:225] Handling call
0426 09:55:26.992457 (+ 242us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:26.993847 (+ 1390us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:26.994090 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:26.994575 10056 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:26.995860 9935 raft_consensus.cc:1226] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 09:55:26.996560 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1051) took 1027 ms. Trace:
I20240426 09:55:26.996670 10056 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:26.996927 9935 rpcz_store.cc:276] 0426 09:55:25.968893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:26.995370 (+1026477us) service_pool.cc:225] Handling call
0426 09:55:26.995658 (+ 288us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:26.996362 (+ 704us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:26.996517 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:26.998191 10056 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:26.999264 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1053) took 1030 ms. Trace:
I20240426 09:55:26.973359 11524 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:27.000329 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:27.001045 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 236689us
I20240426 09:55:27.001394 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:26.976140 11657 raft_consensus.cc:2726] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:27.013073 10056 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.029009 9936 raft_consensus.cc:2379] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:26.980049 11404 leader_election.cc:288] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:26.968873 11680 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:27.031479 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 57651.2us
I20240426 09:55:27.030611 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:27.033787 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:26.973277 10081 rpcz_store.cc:276] 0426 09:55:23.458704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.417548 (+1958844us) service_pool.cc:225] Handling call
0426 09:55:25.864018 (+446470us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.951410 (+ 87392us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:26.960612 (+1009202us) spinlock_profiling.cc:243] Waited 433 ms on lock 0x7b54001815d0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c45b99 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:26.960741 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":996147584}
I20240426 09:55:27.002367 11463 raft_consensus.cc:2781] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:27.037200 11463 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:55:27.036373 11674 log.cc:924] Time spent T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.516s user 0.000s sys 0.005s
I20240426 09:55:27.029342 11524 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:27.038151 11524 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:27.037477 9936 raft_consensus.cc:1261] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:27.038890 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1054) took 1069 ms. Trace:
I20240426 09:55:27.039172 9936 rpcz_store.cc:276] 0426 09:55:25.969353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.036910 (+1067557us) service_pool.cc:225] Handling call
0426 09:55:27.037193 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.038665 (+ 1472us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.038842 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:27.040246 9936 raft_consensus.cc:1226] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:27.040896 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1055) took 1071 ms. Trace:
I20240426 09:55:27.041164 9936 rpcz_store.cc:276] 0426 09:55:25.969499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.039839 (+1070340us) service_pool.cc:225] Handling call
0426 09:55:27.040102 (+ 263us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.040703 (+ 601us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.040853 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:27.042246 10056 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:26.979113 11686 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:27.043066 11686 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.045125 11686 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:27.045500 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:27.002096 11659 raft_consensus.cc:2726] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:27.001940 11654 raft_consensus.cc:2726] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:27.004280 11504 raft_consensus.cc:3032] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:27.047056 11504 raft_consensus.cc:729] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:27.047919 11504 consensus_queue.cc:260] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.002305 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 1047) took 1034 ms. Trace:
I20240426 09:55:27.049134 11504 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:27.049392 9932 rpcz_store.cc:276] 0426 09:55:25.968216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:26.002996 (+ 34780us) service_pool.cc:225] Handling call
0426 09:55:27.000018 (+997022us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:27.050380 11679 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:27.050812 9932 raft_consensus.cc:1226] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:27.051471 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1057) took 1081 ms. Trace:
I20240426 09:55:27.051826 9932 rpcz_store.cc:276] 0426 09:55:25.969792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.050302 (+1080510us) service_pool.cc:225] Handling call
0426 09:55:27.050605 (+ 303us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.051242 (+ 637us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.051421 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:27.053314 9932 raft_consensus.cc:1261] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20240426 09:55:27.053711 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:27.054399 10083 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:27.055665 10083 raft_consensus.cc:1261] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:27.058565 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 988) took 1077 ms. Trace:
I20240426 09:55:27.058977 10009 rpcz_store.cc:276] 0426 09:55:25.981311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.981565 (+ 254us) service_pool.cc:225] Handling call
0426 09:55:25.981811 (+ 246us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:25.981985 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 09:55:26.008191 (+ 26206us) log.cc:842] Serialized 14 byte log entry
0426 09:55:26.008519 (+ 328us) log.cc:842] Serialized 14 byte log entry
0426 09:55:26.520110 (+511591us) spinlock_profiling.cc:243] Waited 2.69 ms on lock 0x7b18003bf4a0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fd1be78 00007f9c9fd1f351 00007f9c9fd1f309 00007f9c9fd1f299 00007f9c9fd1f261 00007f9c9fd1e55d 00007f9ca8f8fd34 00007f9ca8f8fb69 00007f9c9fcfe5f6
0426 09:55:27.058049 (+537939us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b18003bf4a0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fc94faa 00007f9ca9383f25 00007f9ca28d74e3 00007f9ca28cceaa 00007f9ca28cd3fb 00007f9ca2c50a1a 00007f9ca2c6009f 00007f9ca2c60018 00007f9ca2c5ff98
0426 09:55:27.058266 (+ 217us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:27.058310 (+ 44us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:55:27.058350 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.058410 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:27.058511 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11100,"prepare.queue_time_us":98884,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":34170,"raft.queue_time_us":1440,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":153,"spinlock_wait_cycles":55395968}
I20240426 09:55:27.049160 11543 consensus_queue.cc:237] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.036274 11462 raft_consensus.cc:2726] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:27.037436 11424 raft_consensus.cc:2726] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:27.036690 10618 raft_consensus.cc:1261] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240426 09:55:27.067417 11661 consensus_queue.cc:1046] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:27.068334 11657 raft_consensus.cc:3032] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:27.068701 11657 raft_consensus.cc:729] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:27.069443 11657 consensus_queue.cc:260] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:27.071096 11663 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:27.071867 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 78310.8us
I20240426 09:55:27.072675 9904 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:27.036298 11427 consensus_queue.cc:237] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.073974 9904 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:27.004367 9935 rpcz_store.cc:276] 0426 09:55:25.969177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:26.997740 (+1028563us) service_pool.cc:225] Handling call
0426 09:55:26.997974 (+ 234us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:26.998161 (+ 187us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:26.998975 (+ 814us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:26.999022 (+ 47us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:55:26.999055 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:26.999110 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:26.999218 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:27.045619 11686 raft_consensus.cc:2726] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 09:55:27.078450 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 329063us
I20240426 09:55:27.075819 9904 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:27.085376 10085 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:27.087160 10085 raft_consensus.cc:1261] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:27.088338 10085 raft_consensus.cc:1226] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 09:55:27.053663 9932 pending_rounds.cc:76] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9: Aborting all ops after (but not including) 1
I20240426 09:55:27.037550 11463 raft_consensus.cc:3037] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:27.051625 11679 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:27.066381 11545 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.088789 11618 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:27.071895 11682 raft_consensus.cc:2781] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:27.070705 11657 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:27.035692 11404 raft_consensus.cc:2726] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:27.108776 10009 raft_consensus.cc:1261] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:27.110711 9904 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:27.112366 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:27.089217 11519 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:27.089074 11462 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:27.115868 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1063) took 1145 ms. Trace:
I20240426 09:55:27.090283 11623 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:27.129894 11462 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.131917 11462 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:27.128283 9940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 171) took 1161 ms. Trace:
I20240426 09:55:27.132191 11513 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:27.116461 9932 pending_rounds.cc:97] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240426 09:55:27.132946 9932 raft_consensus.cc:2866] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:27.132488 9940 rpcz_store.cc:276] 0426 09:55:25.967108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.967228 (+ 120us) service_pool.cc:225] Handling call
0426 09:55:27.127625 (+1160397us) spinlock_profiling.cc:243] Waited 6.88 ms on lock 0x7b5800004598. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:55:27.127797 (+ 172us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.127948 (+ 151us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:27.127984 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:27.128020 (+ 36us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:55:27.128047 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.128096 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:27.128215 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15828224}
I20240426 09:55:27.135169 11682 raft_consensus.cc:686] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:27.136126 11682 consensus_queue.cc:237] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.136015 9940 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:27.138331 9940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1064) took 1167 ms. Trace:
I20240426 09:55:27.138605 9940 rpcz_store.cc:276] 0426 09:55:25.970875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.134596 (+1163721us) service_pool.cc:225] Handling call
0426 09:55:27.134984 (+ 388us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.138120 (+ 3136us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.138287 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:27.140022 9940 raft_consensus.cc:1261] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:27.140475 9940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1065) took 1169 ms. Trace:
I20240426 09:55:27.140750 9940 rpcz_store.cc:276] 0426 09:55:25.971245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.139403 (+1168158us) service_pool.cc:225] Handling call
0426 09:55:27.139663 (+ 260us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.140285 (+ 622us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.140435 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:27.141976 9940 raft_consensus.cc:1226] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 09:55:27.142470 9940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1235) took 1159 ms. Trace:
I20240426 09:55:27.142748 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1240) took 1065 ms. Trace:
I20240426 09:55:27.143168 9934 rpcz_store.cc:276] 0426 09:55:26.077289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:26.133453 (+ 56164us) service_pool.cc:225] Handling call
0426 09:55:27.142533 (+1009080us) spinlock_profiling.cc:243] Waited 584 ms on lock 0x7b5400220bd4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:27.142670 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1343194496}
I20240426 09:55:27.143715 11680 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:27.144533 9934 raft_consensus.cc:1226] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20240426 09:55:27.105105 11230 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:27.145346 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1236) took 1145 ms. Trace:
I20240426 09:55:27.145054 11230 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.145788 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:27.147500 11230 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:27.090235 11582 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:27.149438 11582 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.091557 11444 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20240426 09:55:27.078495 11424 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:27.149004 9935 rpcz_store.cc:276] 0426 09:55:25.970684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.077274 (+1106590us) service_pool.cc:225] Handling call
0426 09:55:27.077534 (+ 260us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.109987 (+ 32453us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:27.110015 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:27.115624 (+ 5609us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:27.115659 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.115715 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:27.115818 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32095,"raft.queue_time_us":32674,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":10107}
I20240426 09:55:27.156020 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 172) took 1006 ms. Trace:
I20240426 09:55:27.089049 11575 raft_consensus.cc:2726] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:27.148921 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1056) took 1179 ms. Trace:
I20240426 09:55:27.142740 9940 rpcz_store.cc:276] 0426 09:55:25.983391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.141351 (+1157960us) service_pool.cc:225] Handling call
0426 09:55:27.141638 (+ 287us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:27.142276 (+ 638us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.142430 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:27.157362 9936 rpcz_store.cc:276] 0426 09:55:25.969653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.041774 (+1072121us) service_pool.cc:225] Handling call
0426 09:55:27.042025 (+ 251us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.046918 (+ 4893us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:27.046945 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:27.046983 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:27.047009 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.148714 (+101705us) spinlock_profiling.cc:243] Waited 4.59 ms on lock 0x7b5400238cd4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:27.148758 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:27.148870 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10546048}
I20240426 09:55:27.123550 9904 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:27.089077 11654 raft_consensus.cc:2726] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 09:55:27.162575 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:27.120535 11661 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:27.156469 9935 rpcz_store.cc:276] 0426 09:55:26.149814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.155196 (+1005382us) service_pool.cc:225] Handling call
0426 09:55:27.155581 (+ 385us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.155732 (+ 151us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:27.155758 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:27.155795 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:27.155824 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.155872 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:27.155973 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:27.133631 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1061) took 1163 ms. Trace:
I20240426 09:55:27.148593 9934 rpcz_store.cc:276] 0426 09:55:25.999954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.144022 (+1144068us) service_pool.cc:225] Handling call
0426 09:55:27.144330 (+ 308us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:27.145117 (+ 787us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.145301 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:27.129011 11644 raft_consensus.cc:2726] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:27.169389 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:27.170706 9936 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:55:27.171312 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1250) took 1006 ms. Trace:
I20240426 09:55:27.171605 9936 rpcz_store.cc:276] 0426 09:55:26.164998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.160197 (+995199us) service_pool.cc:225] Handling call
0426 09:55:27.170438 (+ 10241us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.171095 (+ 657us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.171264 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:27.183070 9940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1256) took 1008 ms. Trace:
I20240426 09:55:27.183509 9940 rpcz_store.cc:276] 0426 09:55:26.174337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.162013 (+987676us) service_pool.cc:225] Handling call
0426 09:55:27.182431 (+ 20418us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.182701 (+ 270us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:27.182750 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:27.182821 (+ 71us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:27.182875 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.182922 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:27.183020 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:27.157610 11653 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:27.059163 11454 consensus_queue.cc:260] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.163821 9904 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:27.166544 9935 raft_consensus.cc:1226] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:27.210021 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1259) took 1017 ms. Trace:
I20240426 09:55:27.210403 9935 rpcz_store.cc:276] 0426 09:55:26.192832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.166067 (+973235us) service_pool.cc:225] Handling call
0426 09:55:27.166394 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.209756 (+ 43362us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.209955 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:55:27.215075 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 51337.8us
I20240426 09:55:27.120685 11679 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.166752 11661 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:27.230572 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1262) took 1006 ms. Trace:
I20240426 09:55:27.236357 9936 rpcz_store.cc:276] 0426 09:55:26.224498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.172271 (+947773us) service_pool.cc:225] Handling call
0426 09:55:27.230098 (+ 57827us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.230285 (+ 187us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:27.230309 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:27.230348 (+ 39us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:27.230377 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.230424 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:27.230521 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:27.241066 9935 raft_consensus.cc:1226] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:27.160705 11424 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:27.260735 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 44903.9us
I20240426 09:55:27.164224 11524 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.227658 11644 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:27.152904 11618 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:27.160194 11519 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:27.266876 10013 raft_consensus.cc:1226] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:27.269867 9982 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:27.160836 11623 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:27.179961 11381 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.230022 11582 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:27.167676 9932 rpcz_store.cc:276] 0426 09:55:25.970379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.052637 (+1082258us) service_pool.cc:225] Handling call
0426 09:55:27.052949 (+ 312us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.133392 (+ 80443us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.133575 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:27.270879 10056 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.233582 11454 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:27.242218 11575 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:27.319010 11575 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:27.321115 11575 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:27.238924 11661 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:27.260280 11654 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:27.306360 11543 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:27.261399 11695 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:27.311424 10009 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20240426 09:55:27.316133 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 45324.5us
I20240426 09:55:27.340654 10056 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.316305 11517 consensus_queue.cc:1046] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:27.316619 11678 raft_consensus.cc:2726] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:27.316931 11230 consensus_queue.cc:1046] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:27.317524 11422 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:27.276872 11644 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 09:55:27.329675 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 90948.4us
I20240426 09:55:27.315807 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:27.341636 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:27.282816 11659 raft_consensus.cc:2726] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:27.327016 10081 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:27.323058 9934 raft_consensus.cc:1261] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:27.274354 11444 consensus_queue.cc:1035] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20240426 09:55:27.225895 11653 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:27.328954 11402 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:27.352300 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 45758.2us
I20240426 09:55:27.345361 10056 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:27.338568 11543 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:27.323208 11607 consensus_queue.cc:1035] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20240426 09:55:27.360460 10056 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.350494 11644 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:27.350847 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:27.367666 10056 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.342485 11582 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20240426 09:55:27.355132 11545 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:27.350713 11696 raft_consensus.cc:2726] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:27.375452 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:27.357964 11678 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:27.361267 11543 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:27.378732 11462 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.002s
I20240426 09:55:27.381999 11519 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:27.364828 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1263) took 1137 ms. Trace:
I20240426 09:55:27.382982 9934 rpcz_store.cc:276] 0426 09:55:26.227589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.189620 (+962031us) service_pool.cc:225] Handling call
0426 09:55:27.261045 (+ 71425us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.351989 (+ 90944us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.362791 (+ 10802us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:27.383607 11463 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.385428 11463 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:27.386054 11653 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:27.391405 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:27.392609 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:27.393147 10081 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:27.394217 10081 raft_consensus.cc:1226] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20240426 09:55:27.395712 11679 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:27.373659 11427 consensus_queue.cc:1035] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20240426 09:55:27.338129 11654 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.346300 11695 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.366611 11336 raft_consensus.cc:3037] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:27.402181 9904 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:27.399129 11654 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:27.425330 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 63767.4us
I20240426 09:55:27.359025 11402 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:27.379506 11678 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:27.421780 9938 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:27.395743 11618 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:27.384107 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.390056 11545 consensus_queue.cc:1035] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20240426 09:55:27.424944 11609 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:27.460060 11402 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:27.457211 11424 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:27.429718 11320 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:27.429872 11678 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:27.466969 11695 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:27.422801 11679 consensus_queue.cc:1035] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20240426 09:55:27.470896 11381 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:27.471289 11230 raft_consensus.cc:3032] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:27.471678 11230 raft_consensus.cc:729] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:27.472551 9939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1050) took 1503 ms. Trace:
I20240426 09:55:27.472940 9939 rpcz_store.cc:276] 0426 09:55:25.968719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:26.987667 (+1018948us) service_pool.cc:225] Handling call
0426 09:55:26.987924 (+ 257us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:27.002985 (+ 15061us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:27.003029 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:27.030527 (+ 27498us) log.cc:842] Serialized 26 byte log entry
0426 09:55:27.279833 (+249306us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:27.279869 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.279938 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:27.472348 (+192410us) raft_consensus.cc:1671] finished
0426 09:55:27.472364 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:27.472493 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14700,"raft.queue_time_us":40972,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":40370,"thread_start_us":181698,"threads_started":1,"wal-append.queue_time_us":350629}
I20240426 09:55:27.470700 11405 raft_consensus.cc:3032] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:27.479574 11405 raft_consensus.cc:729] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:27.480015 9940 raft_consensus.cc:1261] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:27.480661 9940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1264) took 1238 ms. Trace:
I20240426 09:55:27.480338 11405 consensus_queue.cc:260] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.481535 11405 raft_consensus.cc:3037] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
W20240426 09:55:27.461369 9937 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.028s sys 0.083s
W20240426 09:55:27.483438 9937 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.690s user 0.028s sys 0.083s
I20240426 09:55:27.483995 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.484731 9935 raft_consensus.cc:2454] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
I20240426 09:55:27.485663 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.474648 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.490002 10056 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.492590 11381 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:27.493152 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:27.491153 10586 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:27.472440 11230 consensus_queue.cc:260] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.481014 9940 rpcz_store.cc:276] 0426 09:55:26.242336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.479219 (+1236883us) service_pool.cc:225] Handling call
0426 09:55:27.479612 (+ 393us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.480399 (+ 787us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.480607 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:27.496433 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.493084 11381 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.495609 11230 raft_consensus.cc:3037] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
W20240426 09:55:27.487525 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 118724us
I20240426 09:55:27.494438 10088 raft_consensus.cc:1226] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:27.486748 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.501680 9935 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:27.502468 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:27.502691 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:27.503772 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.457440 11524 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:27.463361 11609 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 09:55:27.506469 11711 log.cc:924] Time spent T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20240426 09:55:27.499774 11457 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:27.504635 10088 raft_consensus.cc:1261] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:27.539567 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:27.546139 10088 raft_consensus.cc:2362] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4. Re-sending same reply.
I20240426 09:55:27.526103 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.528592 10583 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:27.497386 11685 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:27.462874 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:27.461328 11661 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.544277 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:27.555671 9979 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.557348 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.565920 9906 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:27.535300 10055 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:27.540120 11623 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:27.544844 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:27.461867 11320 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.500924 11621 raft_consensus.cc:2781] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:27.526749 9939 raft_consensus.cc:2362] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4. Re-sending same reply.
I20240426 09:55:27.501919 11679 raft_consensus.cc:2726] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:27.526170 10090 raft_consensus.cc:1261] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20240426 09:55:27.604955 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 99584.3us
I20240426 09:55:27.604985 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.549041 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:27.466099 9934 raft_consensus.cc:2454] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:27.500468 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:27.507947 11600 consensus_queue.cc:1046] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.025s
I20240426 09:55:27.609252 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:27.599920 11473 consensus_queue.cc:1035] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:27.555799 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:27.533897 11609 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:27.587095 10008 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:27.637723 10056 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:27.597090 9979 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.598348 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.587494 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:27.589682 10009 raft_consensus.cc:1226] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:27.595587 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:27.600764 10013 raft_consensus.cc:1226] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240426 09:55:27.567364 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:27.601696 9936 raft_consensus.cc:1226] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:27.602550 10089 raft_consensus.cc:1226] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:27.602034 10085 raft_consensus.cc:1261] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:55:27.606405 10090 raft_consensus.cc:1261] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240426 09:55:27.606719 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:27.509388 11695 consensus_queue.cc:1035] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20240426 09:55:27.607623 11621 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:27.650096 10007 raft_consensus.cc:2396] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 09:55:27.605670 11381 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:27.608775 9982 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:27.544438 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:27.609763 11536 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:27.653229 10085 raft_consensus.cc:1261] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 09:55:27.606666 9939 raft_consensus.cc:2454] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:27.554585 11618 raft_consensus.cc:2781] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:27.539595 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:27.636216 11698 log.cc:924] Time spent T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20240426 09:55:27.616533 10088 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
I20240426 09:55:27.639374 10008 raft_consensus.cc:595] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.653410 10090 pending_rounds.cc:76] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee: Aborting all ops after (but not including) 2
W20240426 09:55:27.608690 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 73485.8us
I20240426 09:55:27.685001 10090 pending_rounds.cc:97] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240426 09:55:27.685339 10090 raft_consensus.cc:2866] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:27.509588 9940 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:27.557102 11490 consensus_queue.cc:1035] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20240426 09:55:27.609431 11652 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:27.685097 11519 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:27.690805 11519 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.690665 11652 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:27.692840 11519 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:27.559175 11685 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:27.693316 11504 consensus_meta.cc:287] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.604s user 0.006s sys 0.000s
I20240426 09:55:27.693989 11652 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:27.693857 11504 consensus_queue.cc:1035] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:27.695446 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:27.695855 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.696040 10614 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:27.697018 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.551312 11457 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:27.655500 10013 raft_consensus.cc:1261] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240426 09:55:27.677199 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:55:27.610173 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 44337.5us
I20240426 09:55:27.699561 9906 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:27.700642 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:27.654253 10089 raft_consensus.cc:1261] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:55:27.655541 10056 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.678885 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:27.617846 10081 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:27.614202 10014 raft_consensus.cc:1226] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:27.677309 11424 raft_consensus.cc:2781] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:27.617081 11653 raft_consensus.cc:2726] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:27.686532 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:27.653048 11721 raft_consensus.cc:2781] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:27.688640 11435 consensus_queue.cc:1035] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:27.689728 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:27.697365 9905 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:27.641634 11524 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20240426 09:55:27.653906 11545 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:27.703673 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:27.721205 10089 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:27.721836 9904 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:27.653492 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.699081 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:27.683866 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:27.705003 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:27.702016 11618 raft_consensus.cc:686] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 4 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:27.712611 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.716254 9906 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:27.702945 11723 consensus_queue.cc:1035] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20240426 09:55:27.717144 11679 raft_consensus.cc:2726] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:27.722726 9935 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
I20240426 09:55:27.737906 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:27.714162 11657 consensus_meta.cc:287] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.556s user 0.007s sys 0.003s
I20240426 09:55:27.725601 9939 raft_consensus.cc:2362] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3. Re-sending same reply.
I20240426 09:55:27.740861 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.741695 9939 raft_consensus.cc:2396] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 09:55:27.723515 11721 raft_consensus.cc:686] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:27.702839 11427 consensus_queue.cc:1035] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20240426 09:55:27.655289 11621 raft_consensus.cc:3037] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:27.758452 10011 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:27.762063 10011 raft_consensus.cc:595] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.762319 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:27.703615 10055 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.682477 11536 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.742477 9906 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:27.725303 11519 raft_consensus.cc:2781] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:27.703323 11452 raft_consensus.cc:2726] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:27.707796 11607 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:27.763427 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 59872us
I20240426 09:55:27.723774 11582 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20240426 09:55:27.763386 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:27.777240 9935 raft_consensus.cc:2396] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 09:55:27.756937 11521 raft_consensus.cc:2726] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:27.721123 11600 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.037s
I20240426 09:55:27.743304 11473 raft_consensus.cc:2781] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:27.771911 9906 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:27.783421 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 174706us
I20240426 09:55:27.774665 10055 leader_election.cc:302] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:27.765022 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.756758 11578 raft_consensus.cc:2781] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:27.778059 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:27.767740 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:27.772673 11504 raft_consensus.cc:2726] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:27.780599 10009 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
W20240426 09:55:27.761309 11718 log.cc:924] Time spent T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.236s user 0.000s sys 0.001s
I20240426 09:55:27.763656 11424 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:27.777628 11661 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:27.771178 11697 raft_consensus.cc:2726] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:27.767549 11519 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 09:55:27.779692 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 124218us
I20240426 09:55:27.788970 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.798321 9935 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:27.787495 10082 raft_consensus.cc:1261] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:27.789515 11578 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:27.790912 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.799657 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:27.788681 10055 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.784468 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1042) took 1817 ms. Trace:
I20240426 09:55:27.776014 11653 consensus_queue.cc:1046] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20240426 09:55:27.803475 10056 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.785542 9982 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:27.790261 11688 raft_consensus.cc:2726] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:27.782961 11607 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.793992 11424 raft_consensus.cc:3037] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:27.811383 9935 raft_consensus.cc:1261] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240426 09:55:27.796104 11575 raft_consensus.cc:2726] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:27.815980 9982 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:27.799901 11519 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:27.811774 11578 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:27.821659 9982 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:27.823313 10056 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.824538 10055 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.825773 11688 raft_consensus.cc:2726] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:27.826225 11729 raft_consensus.cc:2726] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:27.826632 11678 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.827286 10056 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.827553 10055 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.828554 11678 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:27.828697 11688 raft_consensus.cc:2726] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:27.828555 10056 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.829730 11688 raft_consensus.cc:2726] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:27.830093 10056 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
W20240426 09:55:27.830740 11454 consensus_meta.cc:287] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.512s user 0.006s sys 0.000s
I20240426 09:55:27.830937 11688 raft_consensus.cc:2726] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:27.831316 10056 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.832175 11688 raft_consensus.cc:2726] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:27.832767 11688 consensus_queue.cc:1035] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:27.834450 11336 raft_consensus.cc:2726] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:27.835211 10055 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.835687 11688 consensus_queue.cc:1035] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:27.791291 11618 consensus_queue.cc:237] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:27.842509 10014 raft_consensus.cc:1261] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:27.850992 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:27.841565 10086 raft_consensus.cc:2396] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 09:55:27.820683 10082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 931) took 1047 ms. Trace:
I20240426 09:55:27.838269 11543 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.858891 10583 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.864122 10583 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:27.859115 10089 raft_consensus.cc:1226] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 1. Current term is 2. Ops: []
I20240426 09:55:27.860040 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:27.791419 11721 consensus_queue.cc:237] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.823834 11697 raft_consensus.cc:2726] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:27.858093 11454 consensus_queue.cc:1046] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20240426 09:55:27.823707 11728 raft_consensus.cc:2726] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:27.855553 11462 raft_consensus.cc:2726] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:27.857947 11733 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:27.855294 11723 raft_consensus.cc:2781] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:27.896121 10011 raft_consensus.cc:1226] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
W20240426 09:55:27.907119 11727 log.cc:924] Time spent T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20240426 09:55:27.855612 9934 raft_consensus.cc:2379] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:27.891053 11416 raft_consensus.cc:2781] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:27.914036 11416 raft_consensus.cc:686] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 4 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:27.914223 11678 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:27.914703 11678 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:27.914901 11416 consensus_queue.cc:237] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:27.915859 11543 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:27.891237 11417 raft_consensus.cc:2781] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:27.862164 10082 rpcz_store.cc:276] 0426 09:55:26.772197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.728638 (+956441us) service_pool.cc:225] Handling call
0426 09:55:27.761618 (+ 32980us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.809385 (+ 47767us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.819361 (+ 9976us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:55:27.931221 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 98873.9us
I20240426 09:55:27.946708 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:27.910224 11609 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:27.906656 11726 log.cc:924] Time spent T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
W20240426 09:55:27.905761 11644 consensus_meta.cc:287] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.531s user 0.008s sys 0.002s
I20240426 09:55:27.952370 10056 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:27.917560 11678 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:27.917821 11536 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:27.920468 11417 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:27.961154 10089 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1102) took 1254 ms. Trace:
I20240426 09:55:27.920195 11723 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:28.006278 9939 raft_consensus.cc:2379] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
W20240426 09:55:28.006237 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 55390.4us
I20240426 09:55:28.003921 10089 rpcz_store.cc:276] 0426 09:55:26.706255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.721913 (+1015658us) service_pool.cc:225] Handling call
0426 09:55:27.722153 (+ 240us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.865614 (+143461us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.960870 (+ 95256us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b5400125554. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:27.961074 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":314603776}
I20240426 09:55:28.011377 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:27.919589 11733 raft_consensus.cc:686] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
W20240426 09:55:28.014322 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 115777us
I20240426 09:55:28.016199 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:27.913946 11578 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.016954 10089 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:28.013345 11723 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:27.971312 11734 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:28.023237 11734 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:28.025138 11734 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:28.026075 11543 raft_consensus.cc:979] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:28.026515 11732 raft_consensus.cc:1067] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Signalling peer f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:27.964148 11644 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.030179 11644 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:28.018467 11417 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:27.814694 9937 rpcz_store.cc:276] 0426 09:55:25.967051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:25.972060 (+ 5009us) service_pool.cc:225] Handling call
0426 09:55:26.792081 (+820021us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b5800004598. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:55:26.792262 (+ 181us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:26.792594 (+ 332us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:26.792628 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:26.793083 (+ 455us) log.cc:842] Serialized 26 byte log entry
0426 09:55:27.483894 (+690811us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:27.484585 (+ 691us) log.cc:842] Serialized 14 byte log entry
0426 09:55:27.484775 (+ 190us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.484857 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:27.772235 (+287378us) raft_consensus.cc:1671] finished
0426 09:55:27.772346 (+ 111us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:27.783016 (+ 10670us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":167,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":330,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":210,"spinlock_wait_cycles":2758912,"thread_start_us":690473,"threads_started":1,"wal-append.queue_time_us":706016}
I20240426 09:55:28.033299 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:28.034276 11723 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:28.036577 9935 raft_consensus.cc:2379] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:28.036614 11723 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:28.037930 11417 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:28.040606 11417 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:28.041793 11609 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:28.043008 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:28.043867 11578 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:28.046797 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:28.048991 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:28.053026 11545 raft_consensus.cc:2726] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:27.962870 11461 consensus_queue.cc:1035] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.091s
I20240426 09:55:28.054458 10086 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 932) took 1266 ms. Trace:
I20240426 09:55:28.054819 10086 rpcz_store.cc:276] 0426 09:55:26.788078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.028736 (+240658us) service_pool.cc:225] Handling call
0426 09:55:28.054290 (+1025554us) spinlock_profiling.cc:243] Waited 807 ms on lock 0x7b540037e954. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:28.054395 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1855985920}
I20240426 09:55:28.055837 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:28.056792 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:28.057674 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:28.019804 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:28.059448 9934 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:28.060326 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:28.060740 10084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1084) took 1400 ms. Trace:
I20240426 09:55:28.060853 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 130) took 1403 ms. Trace:
I20240426 09:55:28.061939 10083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1089) took 1387 ms. Trace:
I20240426 09:55:28.062425 11607 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:27.972379 11688 raft_consensus.cc:2726] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:28.047039 9940 raft_consensus.cc:2379] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:28.061343 10087 rpcz_store.cc:276] 0426 09:55:26.657026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:26.657131 (+ 105us) service_pool.cc:225] Handling call
0426 09:55:26.657427 (+ 296us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:27.105327 (+447900us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:27.105356 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:27.122569 (+ 17213us) log.cc:842] Serialized 26 byte log entry
0426 09:55:27.259166 (+136597us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:55:28.060304 (+801138us) log.cc:842] Serialized 14 byte log entry
0426 09:55:28.060529 (+ 225us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:28.060617 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:28.060660 (+ 43us) raft_consensus.cc:1671] finished
0426 09:55:28.060672 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:28.060805 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":796986,"prepare.queue_time_us":795199,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":24,"raft.queue_time_us":131600,"raft.run_cpu_time_us":443,"raft.run_wall_time_us":3027,"thread_start_us":540608,"threads_started":2,"wal-append.queue_time_us":212616}
I20240426 09:55:28.073239 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:28.064349 10055 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:28.073807 10087 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:28.074972 10056 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:28.058552 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:28.078421 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:28.079069 10089 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:28.077520 10086 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:28.085561 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:28.062541 11691 raft_consensus.cc:2726] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:28.061828 11463 raft_consensus.cc:2726] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:28.087502 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:28.093560 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:28.093919 10088 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1099) took 1388 ms. Trace:
I20240426 09:55:28.095115 10087 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 5.
I20240426 09:55:28.094270 10086 raft_consensus.cc:2379] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 5: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:28.094281 10088 rpcz_store.cc:276] 0426 09:55:26.705557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.683016 (+977459us) service_pool.cc:225] Handling call
0426 09:55:27.683297 (+ 281us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:27.683550 (+ 253us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:27.683566 (+ 16us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:27.688419 (+ 4853us) log.cc:842] Serialized 26 byte log entry
0426 09:55:27.726338 (+ 37919us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:55:28.063120 (+336782us) log.cc:842] Serialized 14 byte log entry
0426 09:55:28.063318 (+ 198us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:28.063382 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:28.093734 (+ 30352us) raft_consensus.cc:1671] finished
0426 09:55:28.093753 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:28.093869 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":336501,"prepare.queue_time_us":336689,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":13815,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":11552,"thread_start_us":37620,"threads_started":1,"wal-append.queue_time_us":102978}
I20240426 09:55:28.034353 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:28.096305 10583 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:28.060940 9934 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:28.096796 10088 raft_consensus.cc:1261] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:28.105705 10088 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 935) took 1274 ms. Trace:
I20240426 09:55:28.085044 11462 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:28.103578 10086 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 939) took 1271 ms. Trace:
I20240426 09:55:28.125236 10086 rpcz_store.cc:276] 0426 09:55:26.832342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.102746 (+1270404us) service_pool.cc:225] Handling call
0426 09:55:28.103007 (+ 261us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:28.103297 (+ 290us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 09:55:28.103322 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:28.103358 (+ 36us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:28.103385 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:28.103444 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:28.103535 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1214,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":72}
W20240426 09:55:28.128628 9938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 1046) took 2160 ms (client timeout 2180 ms). Trace:
W20240426 09:55:28.129004 9938 rpcz_store.cc:269] 0426 09:55:25.968059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:26.240005 (+271946us) service_pool.cc:225] Handling call
0426 09:55:28.128478 (+1888473us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400038ed4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:28.128579 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2639672192}
I20240426 09:55:28.130074 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:28.132122 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1066) took 1440 ms. Trace:
I20240426 09:55:28.105938 11723 consensus_queue.cc:1035] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 09:55:28.132668 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:28.102411 11697 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:28.133527 11697 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.106101 11741 consensus_queue.cc:1035] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.002s
I20240426 09:55:28.062371 10083 rpcz_store.cc:276] 0426 09:55:26.674853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.056408 (+381555us) service_pool.cc:225] Handling call
0426 09:55:27.056728 (+ 320us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:27.057075 (+ 347us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:27.057104 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:27.057543 (+ 439us) log.cc:842] Serialized 26 byte log entry
0426 09:55:27.267157 (+209614us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:28.060902 (+793745us) log.cc:842] Serialized 14 byte log entry
0426 09:55:28.061644 (+ 742us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:28.061717 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:28.061760 (+ 43us) raft_consensus.cc:1671] finished
0426 09:55:28.061775 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:28.061889 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":781342,"prepare.queue_time_us":783586,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":953,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":143,"thread_start_us":180739,"threads_started":1,"wal-append.queue_time_us":308861}
I20240426 09:55:28.138051 10083 raft_consensus.cc:1261] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240426 09:55:28.138958 11688 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:28.139408 11688 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.141285 11688 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:28.143697 11697 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:28.144163 11457 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:28.144920 9839 catalog_manager.cc:5484] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 2 to 3, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 3 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:28.146159 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:28.145766 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:28.110685 11519 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.148625 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:28.149181 9940 raft_consensus.cc:2362] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3. Re-sending same reply.
I20240426 09:55:28.149261 11519 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:28.111946 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:28.149979 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:28.150491 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:28.112912 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:28.151835 9934 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:28.100476 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 938) took 1268 ms. Trace:
I20240426 09:55:28.152235 10087 rpcz_store.cc:276] 0426 09:55:26.832134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.099799 (+1267665us) service_pool.cc:225] Handling call
0426 09:55:28.100072 (+ 273us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:28.100221 (+ 149us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:28.100242 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:28.100277 (+ 35us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:28.100304 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:28.100349 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:28.100428 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:28.153609 10087 raft_consensus.cc:1261] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240426 09:55:28.155282 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:28.114854 10089 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:28.107004 10088 rpcz_store.cc:276] 0426 09:55:26.831441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.096263 (+1264822us) service_pool.cc:225] Handling call
0426 09:55:28.096520 (+ 257us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:28.105466 (+ 8946us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:28.105653 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:28.155947 9932 raft_consensus.cc:1261] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:55:28.156780 10088 raft_consensus.cc:1226] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:28.157610 10083 raft_consensus.cc:1261] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 09:55:28.125953 11678 consensus_queue.cc:1046] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20240426 09:55:28.159432 11691 raft_consensus.cc:3032] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:28.159790 11691 raft_consensus.cc:729] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:28.058534 11621 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:28.117290 11685 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:28.073377 10084 rpcz_store.cc:276] 0426 09:55:26.660229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:26.673745 (+ 13516us) service_pool.cc:225] Handling call
0426 09:55:26.945947 (+272202us) spinlock_profiling.cc:243] Waited 31.5 ms on lock 0x7b58000b0498. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce43ee 00000000003399f9 00007f9ca7b13570 00007f9ca7bd9294 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:55:26.946136 (+ 189us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:26.946549 (+ 413us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:26.946584 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:26.947078 (+ 494us) log.cc:842] Serialized 26 byte log entry
0426 09:55:27.204806 (+257728us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:28.059940 (+855134us) spinlock_profiling.cc:243] Waited 2.57 ms on lock 0x7b18003f7960. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fc94faa 00007f9ca9383f25 00007f9c9fd19094 00007f9c9fd18985 00007f9ca661cc07 00007f9ca2c50997 00007f9ca2c6009f 00007f9ca2c60018 00007f9ca2c5ff98
0426 09:55:28.060196 (+ 256us) log.cc:842] Serialized 14 byte log entry
0426 09:55:28.060406 (+ 210us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:28.060506 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:28.060549 (+ 43us) raft_consensus.cc:1671] finished
0426 09:55:28.060563 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:28.060689 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55736,"prepare.queue_time_us":854907,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":2951,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":182,"spinlock_wait_cycles":78277376,"thread_start_us":256177,"threads_started":1,"wal-append.queue_time_us":295519}
I20240426 09:55:28.083788 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:28.144774 9836 catalog_manager.cc:5484] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index 2 to 3, term changed from 1 to 3, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to <none>, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:28.164682 10011 raft_consensus.cc:1226] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:28.113507 11733 consensus_queue.cc:237] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.118571 11513 raft_consensus.cc:3032] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:28.122561 9982 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:55:28.173398 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 50901.8us
I20240426 09:55:28.173744 10586 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:28.175216 9982 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:28.117189 9979 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:28.124838 11462 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:28.083678 11454 raft_consensus.cc:2781] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:28.146528 9839 catalog_manager.cc:5484] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 3 to 4, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 4 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:28.157984 10088 raft_consensus.cc:1261] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20240426 09:55:28.076833 11424 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.176879 10088 pending_rounds.cc:76] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee: Aborting all ops after (but not including) 2
I20240426 09:55:28.132490 9936 rpcz_store.cc:276] 0426 09:55:26.691502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.237243 (+545741us) service_pool.cc:225] Handling call
0426 09:55:27.268083 (+ 30840us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.654556 (+386473us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:28.131921 (+477365us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b54003eec54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:28.132070 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":766867328}
I20240426 09:55:28.169044 10056 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:28.177157 10088 pending_rounds.cc:97] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20240426 09:55:28.172224 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1100) took 1466 ms. Trace:
I20240426 09:55:28.165810 9836 catalog_manager.cc:5484] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 2, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 2 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:28.176390 11462 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:28.177436 10088 raft_consensus.cc:2866] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:28.177378 9839 catalog_manager.cc:5484] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 1 to 3, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 3 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:28.178323 9836 catalog_manager.cc:5484] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:28.176476 11454 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:28.180271 9836 catalog_manager.cc:5484] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 3, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 3 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:28.181066 11454 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:28.183234 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:28.183868 9936 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:28.184702 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:28.185487 9936 raft_consensus.cc:2396] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 09:55:28.177565 10085 rpcz_store.cc:276] 0426 09:55:26.705790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.687372 (+981582us) service_pool.cc:225] Handling call
0426 09:55:27.707403 (+ 20031us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:27.767373 (+ 59970us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:27.767404 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:27.782609 (+ 15205us) log.cc:842] Serialized 26 byte log entry
0426 09:55:27.893544 (+110935us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:27.893576 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:27.893633 (+ 57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:27.910596 (+ 16963us) raft_consensus.cc:1671] finished
0426 09:55:27.910614 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:28.171994 (+261380us) spinlock_profiling.cc:243] Waited 19.5 ms on lock 0x7b54000daf50. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c45b99 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:28.172169 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":14242,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":3823,"spinlock_wait_cycles":44931840,"thread_start_us":156272,"threads_started":2,"wal-append.queue_time_us":120769}
I20240426 09:55:28.188936 10085 raft_consensus.cc:1261] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:55:28.190613 10085 raft_consensus.cc:1261] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:28.191205 11454 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.193675 10086 raft_consensus.cc:1226] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:28.195304 11454 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:28.243320 10009 raft_consensus.cc:1226] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:28.194975 11462 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.196938 9835 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50252 (request call id 189) took 8815 ms. Trace:
I20240426 09:55:28.197202 9936 raft_consensus.cc:1226] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 09:55:28.193017 10085 raft_consensus.cc:1226] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240426 09:55:28.199100 9836 catalog_manager.cc:5484] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 3, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 3 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:28.151115 9937 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:28.198189 10082 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:55:28.239955 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 70998us
I20240426 09:55:28.241479 9940 raft_consensus.cc:1226] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:28.124452 11740 raft_consensus.cc:2781] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:28.239367 11659 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:28.239161 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 122037us
I20240426 09:55:28.270164 9979 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:28.272145 9979 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:28.173074 11513 raft_consensus.cc:729] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:28.260131 9836 catalog_manager.cc:5484] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 2 to 3, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 3 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:28.261863 9904 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:28.198055 11623 raft_consensus.cc:2726] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:28.258059 11723 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:28.280306 11723 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.262750 10085 raft_consensus.cc:1261] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20240426 09:55:28.262912 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 96865.5us
I20240426 09:55:28.283073 9979 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:28.197902 11408 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:28.265309 10056 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
W20240426 09:55:28.257879 11743 log.cc:924] Time spent T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20240426 09:55:28.160667 11691 consensus_queue.cc:260] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.238837 11427 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:28.242575 11721 raft_consensus.cc:2726] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:28.245436 11519 raft_consensus.cc:2726] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:28.258658 11381 raft_consensus.cc:2781] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:28.294217 10081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1103) took 1587 ms. Trace:
I20240426 09:55:28.247193 9835 rpcz_store.cc:276] 0426 09:55:19.351458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:19.351621 (+ 163us) service_pool.cc:225] Handling call
0426 09:55:28.166818 (+8815197us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:24.722503 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:24.723017 (+ 514us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:24.723052 (+ 35us) write_op.cc:427] Acquired schema lock
0426 09:55:24.723069 (+ 17us) tablet.cc:660] Decoding operations
0426 09:55:24.723361 (+ 292us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:24.723412 (+ 51us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:24.723437 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0426 09:55:24.723638 (+ 201us) tablet.cc:699] Row locks acquired
0426 09:55:24.723655 (+ 17us) write_op.cc:260] PREPARE: finished
0426 09:55:24.723770 (+ 115us) write_op.cc:270] Start()
0426 09:55:24.725013 (+ 1243us) write_op.cc:276] Timestamp: P: 1714125324723730 usec, L: 0
0426 09:55:24.725034 (+ 21us) op_driver.cc:331] REPLICATION: starting
0426 09:55:24.852116 (+127082us) log.cc:842] Serialized 540 byte log entry
0426 09:55:25.247097 (+394981us) op_driver.cc:444] REPLICATION: finished
0426 09:55:25.452867 (+205770us) write_op.cc:301] APPLY: starting
0426 09:55:25.452970 (+ 103us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:25.453075 (+ 105us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:25.453194 (+ 119us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:25.453415 (+ 221us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:25.453579 (+ 164us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0426 09:55:25.453627 (+ 48us) write_op.cc:312] APPLY: finished
0426 09:55:25.453847 (+ 220us) log.cc:842] Serialized 22 byte log entry
0426 09:55:25.453998 (+ 151us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:25.454083 (+ 85us) write_op.cc:446] Released schema lock
0426 09:55:25.454122 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0426 09:55:25.454372 (+ 250us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:25.454397 (+ 25us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":12388,"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":198678,"apply.run_cpu_time_us":1786,"apply.run_wall_time_us":8595,"num_ops":1,"prepare.queue_time_us":92686,"prepare.run_cpu_time_us":20367,"prepare.run_wall_time_us":335879,"raft.queue_time_us":149090,"raft.run_cpu_time_us":8304,"raft.run_wall_time_us":171356,"replication_time_us":521962,"thread_start_us":453512,"threads_started":4,"wal-append.queue_time_us":232214,"wal-append.run_cpu_time_us":44283,"wal-append.run_wall_time_us":1372593}]]}
I20240426 09:55:28.290088 9937 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 5.
I20240426 09:55:28.263967 11680 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:28.292109 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:28.282526 11723 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:28.306857 10081 rpcz_store.cc:276] 0426 09:55:26.706468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.725085 (+1018617us) service_pool.cc:225] Handling call
0426 09:55:27.775109 (+ 50024us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:27.785129 (+ 10020us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 09:55:28.104528 (+319399us) log.cc:842] Serialized 14 byte log entry
0426 09:55:28.293890 (+189362us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:28.293949 (+ 59us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:55:28.293990 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:28.294058 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:28.294165 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":309952,"prepare.queue_time_us":303114,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"thread_start_us":142215,"threads_started":1}
I20240426 09:55:28.323796 10009 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:28.325261 9937 raft_consensus.cc:1226] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240426 09:55:28.260443 11435 consensus_queue.cc:1035] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20240426 09:55:28.286445 10084 raft_consensus.cc:1226] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 09:55:28.278543 11533 raft_consensus.cc:2726] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:28.302021 11408 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:28.290966 11517 raft_consensus.cc:2726] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:28.303543 11461 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:28.266595 11697 consensus_queue.cc:1046] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20240426 09:55:28.331092 11609 consensus_queue.cc:1035] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:28.303336 11644 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election won for term 4
W20240426 09:55:28.332666 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 67431.7us
I20240426 09:55:28.278265 11740 raft_consensus.cc:686] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 5 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:28.290726 11545 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 09:55:28.393177 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 55254.4us
I20240426 09:55:28.395076 10012 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "60bebc132e4141bdbab199326986516e"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:28.395576 10012 raft_consensus.cc:595] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:28.395747 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:28.279785 11659 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:28.339942 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 54638.3us
I20240426 09:55:28.341718 11473 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:28.343605 11320 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:28.346385 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:28.268497 11578 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20240426 09:55:28.407097 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:28.407269 9939 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:28.407702 10081 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:28.378638 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:28.331429 11732 consensus_queue.cc:1035] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:28.303215 11728 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:28.275707 11653 consensus_queue.cc:1035] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.054s
W20240426 09:55:28.381103 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 52318.9us
I20240426 09:55:28.300360 11729 raft_consensus.cc:2726] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:28.337147 11521 consensus_queue.cc:1035] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20240426 09:55:28.396031 11545 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:28.350203 11408 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:28.401757 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:28.276181 11490 consensus_queue.cc:1035] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20240426 09:55:28.380337 11691 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:28.348242 11457 raft_consensus.cc:2726] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:55:28.418129 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:28.424876 9939 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:28.449692 9937 raft_consensus.cc:3037] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:28.450079 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:28.399813 11582 consensus_queue.cc:1046] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20240426 09:55:28.428637 10009 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:28.319885 11543 consensus_queue.cc:1046] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.086s
I20240426 09:55:28.415892 11473 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:28.320236 11427 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.388955 10056 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:28.406751 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:28.330564 11513 consensus_queue.cc:260] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.440901 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:28.468233 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 79361.7us
I20240426 09:55:28.468950 10056 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:28.469991 10056 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:28.416898 11722 consensus_queue.cc:1035] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.041s
I20240426 09:55:28.445005 11545 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:28.442953 11533 consensus_queue.cc:1046] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:28.446761 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:28.405156 11644 raft_consensus.cc:686] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:28.378190 11461 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.420675 11454 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:28.454241 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:28.454345 10055 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:28.456693 10014 raft_consensus.cc:1226] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 09:55:28.403376 11405 consensus_queue.cc:1046] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20240426 09:55:28.460546 9937 raft_consensus.cc:2454] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 5.
I20240426 09:55:28.463958 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:28.525885 10081 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:28.526970 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:28.527458 10081 raft_consensus.cc:3037] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 5
W20240426 09:55:28.473794 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 59596.2us
I20240426 09:55:28.420359 11463 consensus_queue.cc:1046] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20240426 09:55:28.442343 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:28.534708 10007 raft_consensus.cc:1261] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:55:28.447636 11621 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:28.477743 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:28.440243 11721 consensus_queue.cc:1046] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20240426 09:55:28.544301 10081 raft_consensus.cc:2454] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 5.
I20240426 09:55:28.545176 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:28.433389 11728 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:28.518698 11752 log.cc:924] Time spent T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240426 09:55:28.548939 10089 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:28.474961 11462 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:28.476897 11685 raft_consensus.cc:2726] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:28.480170 11336 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:28.551512 10081 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:28.480286 11457 raft_consensus.cc:2726] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:28.559216 10054 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:28.466065 11577 consensus_queue.cc:1035] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.084s
I20240426 09:55:28.558240 11336 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:28.645056 10081 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:28.646143 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:28.647221 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 170196us
I20240426 09:55:28.648180 10056 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:28.552472 11328 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:28.558874 11405 raft_consensus.cc:3032] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:28.562819 11462 raft_consensus.cc:2781] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:28.579566 11685 raft_consensus.cc:2726] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:28.525575 11454 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.662830 11328 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:28.687693 11461 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:28.688103 11659 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:28.689152 11424 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:28.647229 11336 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:28.657220 11405 raft_consensus.cc:729] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:28.708285 11454 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:28.525655 11740 consensus_queue.cc:237] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:28.559971 11739 consensus_queue.cc:1046] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.103s
I20240426 09:55:28.694440 11685 raft_consensus.cc:2726] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:28.710886 11764 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:55:28.731652 11753 log.cc:924] Time spent T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20240426 09:55:28.732903 11764 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:28.734385 11755 log.cc:924] Time spent T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.213s user 0.001s sys 0.000s
I20240426 09:55:28.735194 11427 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:28.735728 11728 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:28.736914 11758 log.cc:924] Time spent T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20240426 09:55:28.742161 11545 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.747210 11678 raft_consensus.cc:3032] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:28.747638 11678 raft_consensus.cc:729] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:28.748113 11545 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:28.748435 11678 consensus_queue.cc:260] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.749508 11474 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:28.749862 11678 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:28.652058 11644 consensus_queue.cc:237] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.756179 11735 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:28.756687 11735 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.783215 11723 raft_consensus.cc:3032] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:28.784281 11723 raft_consensus.cc:729] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:28.785111 11723 consensus_queue.cc:260] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.786118 11723 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:28.789080 11764 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:28.839910 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1079) took 1336 ms. Trace:
I20240426 09:55:28.840782 9936 rpcz_store.cc:276] 0426 09:55:27.504287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.255876 (+751589us) service_pool.cc:225] Handling call
0426 09:55:28.256217 (+ 341us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:28.256419 (+ 202us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:28.256443 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:28.256873 (+ 430us) log.cc:842] Serialized 26 byte log entry
0426 09:55:28.400298 (+143425us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:28.400334 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:28.400398 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:28.839183 (+438785us) raft_consensus.cc:1671] finished
0426 09:55:28.839226 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:28.839585 (+ 359us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":143076,"threads_started":1,"wal-append.queue_time_us":268811}
I20240426 09:55:28.842355 9933 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:28.847558 9940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1080) took 1342 ms. Trace:
I20240426 09:55:28.847995 9940 rpcz_store.cc:276] 0426 09:55:27.504615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.264945 (+760330us) service_pool.cc:225] Handling call
0426 09:55:28.283475 (+ 18530us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:28.299851 (+ 16376us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:28.299880 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:28.321928 (+ 22048us) log.cc:842] Serialized 26 byte log entry
0426 09:55:28.444289 (+122361us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:28.444324 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:28.444386 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:28.847131 (+402745us) raft_consensus.cc:1671] finished
0426 09:55:28.847163 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:28.847380 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16055,"raft.queue_time_us":16752,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":28794,"thread_start_us":47674,"threads_started":1,"wal-append.queue_time_us":199214}
I20240426 09:55:28.851449 10009 raft_consensus.cc:1226] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:28.851917 9940 raft_consensus.cc:1261] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:28.853605 10009 raft_consensus.cc:1261] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20240426 09:55:28.854116 10009 pending_rounds.cc:76] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e: Aborting all ops after (but not including) 5
I20240426 09:55:28.854423 10009 pending_rounds.cc:97] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20240426 09:55:28.854910 10009 raft_consensus.cc:2866] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:28.856807 10010 raft_consensus.cc:1226] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5. Ops: []
W20240426 09:55:28.868657 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 414233us
I20240426 09:55:28.870462 10088 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 956) took 1100 ms. Trace:
I20240426 09:55:28.871047 10088 rpcz_store.cc:276] 0426 09:55:27.770244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.178538 (+408294us) service_pool.cc:225] Handling call
0426 09:55:28.178827 (+ 289us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:28.245428 (+ 66601us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:28.245459 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:28.263650 (+ 18191us) log.cc:842] Serialized 26 byte log entry
0426 09:55:28.388061 (+124411us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:28.438704 (+ 50643us) log.cc:842] Serialized 14 byte log entry
0426 09:55:28.438987 (+ 283us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:28.439052 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:28.677156 (+238104us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x7b1800232840. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fd1be78 00007f9c9fd1f351 00007f9c9fd1f309 00007f9c9fd1f299 00007f9c9fd1f261 00007f9c9fd1e55d 00007f9ca8f8fd34 00007f9ca8f8fb69 00007f9c9fcfe5f6
0426 09:55:28.733814 (+ 56658us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:55:28.869786 (+135972us) raft_consensus.cc:1671] finished
0426 09:55:28.869837 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:28.870169 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":119408,"prepare.queue_time_us":56132,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":86844,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":9770,"spinlock_wait_cycles":9896448,"thread_start_us":86602,"threads_started":1,"wal-append.queue_time_us":171462}
I20240426 09:55:28.658191 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
W20240426 09:55:28.883095 10082 consensus_meta.cc:287] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.603s user 0.008s sys 0.008s
I20240426 09:55:28.888268 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
W20240426 09:55:28.889863 9934 consensus_meta.cc:287] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.737s user 0.009s sys 0.009s
I20240426 09:55:28.890249 9934 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:28.891222 9935 raft_consensus.cc:1226] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240426 09:55:28.893539 11582 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:28.894616 11582 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.897958 11582 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:28.885243 10583 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:28.728672 11462 raft_consensus.cc:686] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:28.901173 11621 raft_consensus.cc:2781] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:28.901791 11621 raft_consensus.cc:686] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [term 5 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:28.887578 10082 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:28.971642 11621 consensus_queue.cc:237] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:28.930652 11328 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:28.976208 11769 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:28.976229 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:28.977310 11769 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:28.979153 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:28.980595 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:28.982244 11771 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:28.981798 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:28.904618 11582 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:28.996832 11582 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.997906 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:28.982933 11771 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:28.984570 11405 consensus_queue.cc:260] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:28.988523 11691 consensus_meta.cc:287] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.536s user 0.005s sys 0.004s
I20240426 09:55:28.901196 11462 consensus_queue.cc:237] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:28.992507 10055 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:28.993213 10082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 957) took 1217 ms. Trace:
W20240426 09:55:28.993430 11408 consensus_meta.cc:287] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.540s user 0.011s sys 0.005s
I20240426 09:55:28.981735 11769 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:29.005995 10082 rpcz_store.cc:276] 0426 09:55:27.778690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.964048 (+185358us) service_pool.cc:225] Handling call
0426 09:55:28.993104 (+1029056us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:55:29.006364 11680 consensus_meta.cc:287] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.635s user 0.006s sys 0.002s
W20240426 09:55:29.025377 9982 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.008s sys 0.010s
W20240426 09:55:29.042980 9982 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.563s user 0.008s sys 0.010s
I20240426 09:55:29.025600 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 1093) took 1005 ms. Trace:
I20240426 09:55:29.004616 11582 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:29.031517 11474 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:29.032001 11774 raft_consensus.cc:2726] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:29.006321 11408 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.064270 10089 raft_consensus.cc:1226] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:29.072384 11408 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:29.072770 11473 consensus_meta.cc:287] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.595s user 0.008s sys 0.004s
I20240426 09:55:29.073271 11473 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:29.076035 11771 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:55:29.077524 9904 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.672s user 0.010s sys 0.011s
I20240426 09:55:29.070214 10009 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:29.077639 11474 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:29.081005 10007 raft_consensus.cc:1226] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:29.083454 10055 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:29.084827 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.086251 11661 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:29.086949 11661 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.088892 11381 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:29.089126 11661 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:29.090497 9904 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.685s user 0.010s sys 0.011s
I20240426 09:55:29.049561 9934 rpcz_store.cc:276] 0426 09:55:28.010510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.112531 (+102021us) service_pool.cc:225] Handling call
0426 09:55:28.983975 (+871444us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:29.068241 9941 raft_consensus.cc:2932] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:29.100052 11771 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:29.100503 11771 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:29.101117 11771 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:29.106834 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:29.108317 11680 raft_consensus.cc:2726] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:29.096697 11381 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:29.120769 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.082634 11474 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:29.148373 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1083) took 1036 ms. Trace:
I20240426 09:55:29.247843 11688 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:29.268569 11688 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:29.269786 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1067) took 2577 ms. Trace:
I20240426 09:55:29.248668 11473 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:29.249513 10088 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 963) took 1173 ms. Trace:
I20240426 09:55:29.250304 10082 raft_consensus.cc:1226] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: [2.2-3.3]
I20240426 09:55:29.100492 11691 consensus_queue.cc:1046] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20240426 09:55:29.255208 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 155013us
W20240426 09:55:29.257252 11336 consensus_meta.cc:287] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.539s user 0.004s sys 0.008s
I20240426 09:55:29.109072 11696 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:29.144054 9979 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.003s sys 0.014s
W20240426 09:55:29.291973 9979 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.785s user 0.005s sys 0.015s
W20240426 09:55:29.292397 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 826362us
I20240426 09:55:29.293780 9979 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:29.295859 11582 consensus_queue.cc:1046] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 09:55:29.292708 11771 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.292002 11696 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.298894 11691 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:29.299880 11696 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:29.299693 11691 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:29.300391 11696 raft_consensus.cc:3032] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Stepping down as leader of term 3
W20240426 09:55:29.275228 11731 log.cc:924] Time spent T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 1.454s user 0.002s sys 0.002s
I20240426 09:55:29.300827 11696 raft_consensus.cc:729] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:29.300267 11723 raft_consensus.cc:2781] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:29.301852 11696 consensus_queue.cc:260] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.302381 11723 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:29.302815 11723 raft_consensus.cc:3037] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:29.305361 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 950) took 1881 ms. Trace:
I20240426 09:55:29.276582 10088 rpcz_store.cc:276] 0426 09:55:28.076309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.874895 (+798586us) service_pool.cc:225] Handling call
0426 09:55:28.875632 (+ 737us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:28.876318 (+ 686us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:28.876350 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:28.876997 (+ 647us) log.cc:842] Serialized 26 byte log entry
0426 09:55:29.026059 (+149062us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:29.078758 (+ 52699us) log.cc:842] Serialized 14 byte log entry
0426 09:55:29.096779 (+ 18021us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.096846 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:29.249237 (+152391us) raft_consensus.cc:1671] finished
0426 09:55:29.249264 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:29.249429 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34015,"prepare.queue_time_us":59695,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":49,"raft.queue_time_us":99339,"raft.run_cpu_time_us":356,"raft.run_wall_time_us":10059,"thread_start_us":148611,"threads_started":1,"wal-append.queue_time_us":181760}
I20240426 09:55:29.275177 10008 raft_consensus.cc:1226] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:29.259487 11774 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:29.266718 9937 rpcz_store.cc:276] 0426 09:55:28.008104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.547252 (+539148us) service_pool.cc:225] Handling call
0426 09:55:28.547523 (+ 271us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:28.839240 (+291717us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:28.839314 (+ 74us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:28.840357 (+ 1043us) log.cc:842] Serialized 26 byte log entry
0426 09:55:29.025751 (+185394us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:29.025782 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.029125 (+ 3343us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:29.029301 (+ 176us) raft_consensus.cc:1671] finished
0426 09:55:29.029317 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:29.044052 (+ 14735us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10767,"raft.queue_time_us":10914,"spinlock_wait_cycles":43652352,"thread_start_us":184895,"threads_started":1,"wal-append.queue_time_us":181847}
I20240426 09:55:29.287345 10081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 962) took 1211 ms. Trace:
I20240426 09:55:29.290906 11336 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.270252 9932 rpcz_store.cc:276] 0426 09:55:26.691776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.333768 (+641992us) service_pool.cc:225] Handling call
0426 09:55:28.155604 (+821836us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:28.156352 (+ 748us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.105748 (+949396us) spinlock_profiling.cc:243] Waited 804 ms on lock 0x7b5400038ed0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c45b99 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:29.269592 (+163844us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1849038592}
I20240426 09:55:29.305732 10087 rpcz_store.cc:276] 0426 09:55:27.424075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.154234 (+730159us) service_pool.cc:225] Handling call
0426 09:55:28.154486 (+ 252us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:28.154797 (+ 311us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:28.154823 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:28.155202 (+ 379us) log.cc:842] Serialized 26 byte log entry
0426 09:55:28.268343 (+113141us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:28.280818 (+ 12475us) log.cc:842] Serialized 14 byte log entry
0426 09:55:29.304224 (+1023406us) spinlock_profiling.cc:243] Waited 3.61 ms on lock 0x7b18000edd00. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fce97c6 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2bda143 00007f9ca2c4709f 00007f9ca2c45b2e
0426 09:55:29.304746 (+ 522us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b54003cfda0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2bda143 00007f9ca2c4709f 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c
0426 09:55:29.304812 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.304948 (+ 136us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:29.305028 (+ 80us) raft_consensus.cc:1671] finished
0426 09:55:29.305039 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:29.305184 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42487,"prepare.queue_time_us":24352,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":46760,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":10924,"spinlock_wait_cycles":37971584,"thread_start_us":112875,"threads_started":1,"wal-append.queue_time_us":121616}
I20240426 09:55:29.302747 11771 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:29.315593 11688 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:29.353826 11783 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:29.354547 11783 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:29.356645 11783 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:29.362903 11504 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:29.363517 11504 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.348306 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:29.365818 10009 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:29.366322 10088 raft_consensus.cc:1261] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240426 09:55:29.345888 10055 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:29.319947 10081 rpcz_store.cc:276] 0426 09:55:28.076081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.657724 (+581643us) service_pool.cc:225] Handling call
0426 09:55:28.679828 (+ 22104us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:28.692375 (+ 12547us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:28.692407 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:28.869216 (+176809us) log.cc:842] Serialized 26 byte log entry
0426 09:55:29.044768 (+175552us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:29.156263 (+111495us) log.cc:842] Serialized 14 byte log entry
0426 09:55:29.257729 (+101466us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.257832 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:29.287010 (+ 29178us) raft_consensus.cc:1671] finished
0426 09:55:29.287028 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:29.287174 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17377,"prepare.queue_time_us":64633,"prepare.run_cpu_time_us":5391,"prepare.run_wall_time_us":103930,"raft.queue_time_us":43556,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":2133,"thread_start_us":172313,"threads_started":1,"wal-append.queue_time_us":404628}
I20240426 09:55:29.352824 9936 raft_consensus.cc:1226] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20240426 09:55:29.369118 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 75502.4us
I20240426 09:55:29.371927 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.376070 10008 raft_consensus.cc:2454] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:29.383428 11774 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.389961 10084 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:29.391027 9982 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:29.393016 11545 raft_consensus.cc:2781] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:29.393607 11545 raft_consensus.cc:686] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:29.394903 11545 consensus_queue.cc:237] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.396298 11782 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:29.397073 11782 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.398742 11646 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:29.399477 11646 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:29.400803 10055 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:29.401722 11771 raft_consensus.cc:2781] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:29.402200 11771 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:29.402575 11771 raft_consensus.cc:3037] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:55:29.402909 11740 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.005s sys 0.016s
W20240426 09:55:29.403298 11740 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s user 0.005s sys 0.016s
I20240426 09:55:29.405265 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1092) took 1395 ms. Trace:
I20240426 09:55:29.405625 9934 rpcz_store.cc:276] 0426 09:55:28.010287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:29.337768 (+1327481us) service_pool.cc:225] Handling call
0426 09:55:29.373545 (+ 35777us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:29.404931 (+ 31386us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:29.404957 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:29.404995 (+ 38us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:55:29.405050 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.405102 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:29.405202 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:29.406726 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:29.380167 11696 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:29.399991 11474 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:29.408156 9979 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:29.371852 11680 consensus_queue.cc:1046] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20240426 09:55:29.415056 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:29.362607 11723 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:29.422969 11723 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:29.423784 11744 consensus_queue.cc:1035] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.006s
I20240426 09:55:29.432830 11691 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:29.434134 11336 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:29.388787 11582 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:29.440526 10081 raft_consensus.cc:1261] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 09:55:29.443450 11729 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:29.451443 11774 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:29.453943 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:29.435766 11740 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:29.437484 11771 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.458006 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:29.456262 11757 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:29.444424 9937 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:47242
I20240426 09:55:29.470338 11517 raft_consensus.cc:2726] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:29.459267 11474 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.470734 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:29.472064 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:29.474424 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 66357.5us
I20240426 09:55:29.475230 9937 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:29.475620 9937 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:29.474040 10009 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:29.477339 10055 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:29.477797 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:29.478464 10081 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:29.479224 10014 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1024) took 1084 ms. Trace:
I20240426 09:55:29.479630 10014 rpcz_store.cc:276] 0426 09:55:28.395058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.395163 (+ 105us) service_pool.cc:225] Handling call
0426 09:55:28.453650 (+ 58487us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:28.523496 (+ 69846us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.478875 (+955379us) spinlock_profiling.cc:243] Waited 58.2 ms on lock 0x7b540016bf50. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c45b99 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:29.479054 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":133864832}
I20240426 09:55:29.481173 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.482632 11771 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:29.482836 11774 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:29.483388 11774 raft_consensus.cc:686] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:29.461907 11740 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:29.484352 11774 consensus_queue.cc:237] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.485320 9979 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:29.486519 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1094) took 1475 ms. Trace:
I20240426 09:55:29.487160 10090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1101) took 2781 ms. Trace:
I20240426 09:55:29.487510 10090 rpcz_store.cc:276] 0426 09:55:26.706027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.689430 (+983403us) service_pool.cc:225] Handling call
0426 09:55:27.689702 (+ 272us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:27.712827 (+ 23125us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:27.712854 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:27.713227 (+ 373us) log.cc:842] Serialized 26 byte log entry
0426 09:55:27.761727 (+ 48500us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:27.761768 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:28.135243 (+373475us) spinlock_profiling.cc:243] Waited 19.5 ms on lock 0x7b54000d9154. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:28.135291 (+ 48us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:29.270496 (+1135205us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:55:29.486839 (+216343us) raft_consensus.cc:1671] finished
0426 09:55:29.486859 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:29.487020 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3283,"raft.queue_time_us":20745,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":12138,"spinlock_wait_cycles":49200640,"thread_start_us":32504,"threads_started":1,"wal-append.queue_time_us":78476}
I20240426 09:55:29.488453 11381 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.488885 11502 raft_consensus.cc:3032] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:29.489324 11502 raft_consensus.cc:729] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:29.490921 11504 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:29.491680 10008 raft_consensus.cc:1226] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:29.492715 11517 raft_consensus.cc:2726] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:29.492916 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:29.493161 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.493723 11416 consensus_queue.cc:1035] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20240426 09:55:29.494005 10015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 1037) took 1089 ms. Trace:
I20240426 09:55:29.496089 11740 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:29.497568 10012 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34938 (request call id 2) took 1103 ms. Trace:
I20240426 09:55:29.497962 10012 rpcz_store.cc:276] 0426 09:55:28.394160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.394952 (+ 792us) service_pool.cc:225] Handling call
0426 09:55:29.497019 (+1102067us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54001ba6a0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2bdbf50 00007f9ca2c40b52 00007f9ca7b8b48a 00007f9ca127acd4 00007f9ca127ac62 00007f9ca127ab91 00007f9ca127ab0c 00007f9ca1279dc2 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:29.497361 (+ 342us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":580537216}
I20240426 09:55:29.499148 11533 raft_consensus.cc:3032] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:29.499576 11533 raft_consensus.cc:729] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:29.500610 11533 consensus_queue.cc:260] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.501768 11533 raft_consensus.cc:3037] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:29.502851 11474 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:29.508234 9904 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:29.510057 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:29.486699 9979 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:29.512943 11772 raft_consensus.cc:2741] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20240426 09:55:29.517370 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 630691us
I20240426 09:55:29.517979 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:29.494197 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:29.472949 11582 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.493516 9939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1304) took 1614 ms. Trace:
I20240426 09:55:29.525188 9934 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:29.494391 10015 rpcz_store.cc:276] 0426 09:55:28.404667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.421311 (+ 16644us) service_pool.cc:225] Handling call
0426 09:55:29.492797 (+1071486us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x7b580004ec98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:55:29.493933 (+ 1136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3145984}
I20240426 09:55:29.526376 11473 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:29.526692 11757 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:29.528306 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:29.521646 9939 rpcz_store.cc:276] 0426 09:55:27.802519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.452343 (+649824us) service_pool.cc:225] Handling call
0426 09:55:28.452620 (+ 277us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:28.525070 (+ 72450us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:28.525097 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:28.525467 (+ 370us) log.cc:842] Serialized 26 byte log entry
0426 09:55:29.019631 (+494164us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:55:29.377580 (+357949us) log.cc:842] Serialized 14 byte log entry
0426 09:55:29.381237 (+ 3657us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.385268 (+ 4031us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:29.388055 (+ 2787us) raft_consensus.cc:1671] finished
0426 09:55:29.388071 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:29.417041 (+ 28970us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":69691,"raft.queue_time_us":76229,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":35701,"thread_start_us":782381,"threads_started":2,"wal-append.queue_time_us":462889}
I20240426 09:55:29.529182 9937 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.533418 10055 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:29.490484 11381 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:29.536219 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:29.544046 10012 raft_consensus.cc:3037] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:29.537902 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.538872 9937 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:29.486929 9932 rpcz_store.cc:276] 0426 09:55:28.010734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:29.378558 (+1367824us) service_pool.cc:225] Handling call
0426 09:55:29.455107 (+ 76549us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:29.461354 (+ 6247us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:29.461977 (+ 623us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:29.473821 (+ 11844us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:29.486263 (+ 12442us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.486342 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:29.486454 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:29.542748 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1085) took 1534 ms. Trace:
I20240426 09:55:29.545524 9936 rpcz_store.cc:276] 0426 09:55:28.008629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.842499 (+833870us) service_pool.cc:225] Handling call
0426 09:55:28.843302 (+ 803us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:29.375026 (+531724us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.542498 (+167472us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b54002620d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:29.542689 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1132865920}
I20240426 09:55:29.528569 11473 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:29.564193 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:47242 (request call id 1096) took 1471 ms. Trace:
I20240426 09:55:29.548902 9941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1306) took 1710 ms. Trace:
I20240426 09:55:29.549542 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:29.570477 9937 rpcz_store.cc:276] 0426 09:55:28.083995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:29.410809 (+1326814us) service_pool.cc:225] Handling call
0426 09:55:29.544866 (+134057us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:29.567864 9941 rpcz_store.cc:276] 0426 09:55:27.838243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.538646 (+700403us) service_pool.cc:225] Handling call
0426 09:55:28.539128 (+ 482us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 09:55:28.555061 (+ 15933us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:28.555089 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 09:55:28.556038 (+ 949us) log.cc:842] Serialized 590 byte log entry
0426 09:55:29.067721 (+511683us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:55:29.529071 (+461350us) log.cc:842] Serialized 14 byte log entry
0426 09:55:29.534387 (+ 5316us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.545473 (+ 11086us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:29.545520 (+ 47us) raft_consensus.cc:1671] finished
0426 09:55:29.545530 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:29.548846 (+ 3316us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17517,"raft.queue_time_us":22880,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":23772,"thread_start_us":178358,"threads_started":1,"wal-append.queue_time_us":475836}
I20240426 09:55:29.576545 10009 raft_consensus.cc:1226] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:29.490408 11502 consensus_queue.cc:260] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.547201 11729 raft_consensus.cc:2726] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:29.552027 11473 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:29.582166 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:29.583658 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:29.579900 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:29.586270 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:29.587159 10014 raft_consensus.cc:2396] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20240426 09:55:29.585372 10055 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.141.130:44411 timed out after 1.652s (SENT) [suppressed 36 similar messages]
I20240426 09:55:29.582077 11504 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:29.594538 10585 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.661s user 0.013s sys 0.054s
W20240426 09:55:29.599061 10585 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.712s user 0.013s sys 0.055s
I20240426 09:55:29.581497 11723 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:29.589344 9904 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
W20240426 09:55:29.599376 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 712556us
I20240426 09:55:29.592653 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:29.605690 11417 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:29.604221 11504 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.634289 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:29.635813 11504 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:29.637153 11416 consensus_queue.cc:1035] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:29.633342 10012 raft_consensus.cc:2454] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:29.640041 10012 raft_consensus.cc:1226] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20240426 09:55:29.640884 11408 raft_consensus.cc:3032] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:29.641307 11408 raft_consensus.cc:729] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:29.641549 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1294) took 2224 ms. Trace:
I20240426 09:55:29.642182 10086 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 943) took 2374 ms. Trace:
I20240426 09:55:29.642563 10086 rpcz_store.cc:276] 0426 09:55:27.268178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.126891 (+858713us) service_pool.cc:225] Handling call
0426 09:55:28.127140 (+ 249us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:28.194027 (+ 66887us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.641831 (+1447804us) spinlock_profiling.cc:243] Waited 66.4 ms on lock 0x7b5400125f54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:29.642044 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":152647040}
I20240426 09:55:29.643741 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:29.643929 11757 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:29.644387 10086 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:55:29.646680 11617 consensus_queue.cc:1046] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 09:55:29.647312 10016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1022) took 1252 ms. Trace:
I20240426 09:55:29.647755 11617 raft_consensus.cc:3032] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:29.648164 11617 raft_consensus.cc:729] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:29.649098 11617 consensus_queue.cc:260] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:29.650400 11617 raft_consensus.cc:3037] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:29.654402 11735 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:29.606732 10011 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.129s user 0.023s sys 0.043s
I20240426 09:55:29.607242 11729 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:29.660485 11729 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.614893 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:29.641887 9935 rpcz_store.cc:276] 0426 09:55:27.417023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.153313 (+736290us) service_pool.cc:225] Handling call
0426 09:55:28.890831 (+737518us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:28.891582 (+ 751us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.641374 (+749792us) spinlock_profiling.cc:243] Waited 737 ms on lock 0x7b54000bf750. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c45b99 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:29.641497 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1695274368}
I20240426 09:55:29.582056 11772 consensus_queue.cc:1035] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.033s
I20240426 09:55:29.615593 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:29.617766 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:29.619990 11740 consensus_queue.cc:1046] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:29.622565 11789 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:29.624719 11474 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:29.629479 9939 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:29.633337 11417 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:29.655314 11723 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:29.605859 11230 consensus_queue.cc:1046] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20240426 09:55:29.606240 11778 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:29.659027 9934 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:29.664278 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:29.647681 10016 rpcz_store.cc:276] 0426 09:55:28.394464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.397541 (+ 3077us) service_pool.cc:225] Handling call
0426 09:55:28.397822 (+ 281us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:28.521281 (+123459us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:28.521314 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:28.521350 (+ 36us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:28.521375 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.646986 (+1125611us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b540025fdd4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:29.647039 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:29.647174 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":283510528}
I20240426 09:55:29.665655 9941 raft_consensus.cc:2396] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20240426 09:55:29.668929 10011 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.222s user 0.024s sys 0.043s
I20240426 09:55:29.676168 9937 raft_consensus.cc:2396] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20240426 09:55:29.650713 11756 log.cc:924] Time spent T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.990s user 0.011s sys 0.027s
I20240426 09:55:29.635728 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:29.680488 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 181) took 1513 ms. Trace:
I20240426 09:55:29.680871 9936 rpcz_store.cc:276] 0426 09:55:28.167492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:29.599555 (+1432063us) service_pool.cc:225] Handling call
0426 09:55:29.669533 (+ 69978us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:29.676720 (+ 7187us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:29.676751 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:29.680251 (+ 3500us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:55:29.680286 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.680340 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:29.680440 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:29.682420 10089 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1119) took 1270 ms. Trace:
I20240426 09:55:29.687772 9934 raft_consensus.cc:1261] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240426 09:55:29.682807 10089 rpcz_store.cc:276] 0426 09:55:28.412185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:29.065798 (+653613us) service_pool.cc:225] Handling call
0426 09:55:29.066194 (+ 396us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:29.066565 (+ 371us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:29.066587 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:29.067060 (+ 473us) log.cc:842] Serialized 26 byte log entry
0426 09:55:29.368537 (+301477us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:29.368568 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.368623 (+ 55us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:29.679622 (+310999us) raft_consensus.cc:1671] finished
0426 09:55:29.679641 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:29.682364 (+ 2723us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":301114,"threads_started":1,"wal-append.queue_time_us":524265}
I20240426 09:55:29.691367 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 183) took 1244 ms. Trace:
I20240426 09:55:29.691758 9934 rpcz_store.cc:276] 0426 09:55:28.444260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:29.679383 (+1235123us) service_pool.cc:225] Handling call
0426 09:55:29.687449 (+ 8066us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:29.688130 (+ 681us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.688289 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:29.686561 11789 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:29.696125 11789 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:29.696795 10082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1126) took 1279 ms. Trace:
I20240426 09:55:29.697170 10082 rpcz_store.cc:276] 0426 09:55:28.417435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:29.297404 (+879969us) service_pool.cc:225] Handling call
0426 09:55:29.297821 (+ 417us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 09:55:29.349836 (+ 52015us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:29.352454 (+ 2618us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 09:55:29.431563 (+ 79109us) log.cc:842] Serialized 52 byte log entry
0426 09:55:29.586345 (+154782us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:29.589041 (+ 2696us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.592301 (+ 3260us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:29.696595 (+104294us) raft_consensus.cc:1671] finished
0426 09:55:29.696612 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:29.696743 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11038,"raft.queue_time_us":39178,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":12750,"thread_start_us":148506,"threads_started":1,"wal-append.queue_time_us":245972}
I20240426 09:55:29.699501 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:29.700263 11778 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:29.702463 11778 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:29.704578 10089 raft_consensus.cc:1226] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240426 09:55:29.705297 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.682716 11474 raft_consensus.cc:686] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:29.712355 9935 raft_consensus.cc:1226] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:29.713712 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1099) took 1370 ms. Trace:
I20240426 09:55:29.714041 10016 raft_consensus.cc:1226] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:29.714097 9935 rpcz_store.cc:276] 0426 09:55:28.342400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:29.667451 (+1325051us) service_pool.cc:225] Handling call
0426 09:55:29.687137 (+ 19686us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:29.712739 (+ 25602us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.712897 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:29.716269 11729 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:29.690256 11582 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:29.719187 11740 consensus_queue.cc:1035] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20240426 09:55:29.722067 11740 consensus_queue.cc:1046] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:29.724036 11417 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:29.724509 11502 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:55:29.729173 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 121146us
I20240426 09:55:29.729861 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.730424 11723 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:29.731153 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:29.721935 10084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1120) took 1308 ms. Trace:
I20240426 09:55:29.732524 11474 consensus_queue.cc:237] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.736428 9935 raft_consensus.cc:1226] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 5. Current term is 6. Ops: []
I20240426 09:55:29.742491 11740 consensus_queue.cc:1046] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:29.733028 10084 rpcz_store.cc:276] 0426 09:55:28.413385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.607086 (+193701us) service_pool.cc:225] Handling call
0426 09:55:29.721711 (+1114625us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b540024ed54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:29.721849 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":346302720}
I20240426 09:55:29.743580 10611 raft_consensus.cc:2396] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240426 09:55:29.744705 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:29.726644 11408 consensus_queue.cc:260] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.725637 11473 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.747323 11408 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:29.751432 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 1117) took 1851 ms. Trace:
I20240426 09:55:29.753060 10008 raft_consensus.cc:1226] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240426 09:55:29.743609 11621 raft_consensus.cc:3032] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:29.752297 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.753070 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.758477 10087 raft_consensus.cc:1226] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240426 09:55:29.755085 10085 rpcz_store.cc:276] 0426 09:55:27.898820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.283928 (+385108us) service_pool.cc:225] Handling call
0426 09:55:28.285362 (+ 1434us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:28.330602 (+ 45240us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:28.330628 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:28.331021 (+ 393us) log.cc:842] Serialized 26 byte log entry
0426 09:55:28.438008 (+106987us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:28.438044 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:28.438104 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:29.650037 (+1211933us) spinlock_profiling.cc:243] Waited 29.1 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:55:29.737889 (+ 87852us) raft_consensus.cc:1671] finished
0426 09:55:29.737909 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:29.750656 (+ 12747us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37804,"raft.queue_time_us":45991,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"spinlock_wait_cycles":67042688,"thread_start_us":106739,"threads_started":1,"wal-append.queue_time_us":262697}
I20240426 09:55:29.759671 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.762485 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:29.765831 9933 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 1060) took 3795 ms (client timeout 1904 ms). Trace:
W20240426 09:55:29.766197 9933 rpcz_store.cc:269] 0426 09:55:25.970210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:26.151546 (+181336us) service_pool.cc:225] Handling call
0426 09:55:29.765628 (+3614082us) spinlock_profiling.cc:243] Waited 2.67 s on lock 0x7b54003769d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:29.765756 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6147075968}
I20240426 09:55:29.767347 9940 raft_consensus.cc:1261] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:55:29.768922 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.769593 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.770184 10008 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:29.756610 11621 raft_consensus.cc:729] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:29.773867 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.776577 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:29.777807 10015 raft_consensus.cc:3032] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:29.778121 10015 raft_consensus.cc:729] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:29.779361 11679 raft_consensus.cc:3032] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:29.779744 11679 raft_consensus.cc:729] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:29.779897 11623 raft_consensus.cc:3032] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:29.780251 11623 raft_consensus.cc:729] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:29.780602 11679 consensus_queue.cc:260] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.781028 11623 consensus_queue.cc:260] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:29.781625 11679 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:29.782400 11623 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Advancing to term 3
W20240426 09:55:29.782877 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 78558.3us
I20240426 09:55:29.778882 10015 consensus_queue.cc:260] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:29.785079 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 199723us
I20240426 09:55:29.789953 10088 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:59730 (request call id 714) took 2627 ms. Trace:
I20240426 09:55:29.790331 10088 rpcz_store.cc:276] 0426 09:55:27.162624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:29.788574 (+2625950us) service_pool.cc:225] Handling call
0426 09:55:29.789866 (+ 1292us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:29.791741 10014 raft_consensus.cc:1261] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240426 09:55:29.794591 10583 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
W20240426 09:55:29.798528 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 113694us
I20240426 09:55:29.807214 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.807749 10014 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:29.808373 10014 raft_consensus.cc:595] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:29.808727 10014 raft_consensus.cc:603] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 09:55:29.803669 11740 raft_consensus.cc:2781] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:29.809433 11740 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:29.809800 11740 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:29.810276 11536 raft_consensus.cc:3032] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:29.810657 11536 raft_consensus.cc:729] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:29.811650 11536 consensus_queue.cc:260] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.812852 9904 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:29.813769 9904 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:29.814594 10088 raft_consensus.cc:1226] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240426 09:55:29.812140 11807 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:29.813021 11536 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
W20240426 09:55:29.816526 11754 log.cc:924] Time spent T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 1.174s user 0.002s sys 0.000s
I20240426 09:55:29.824496 10087 raft_consensus.cc:1226] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:29.824501 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.825541 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.825991 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:29.826246 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1300) took 2336 ms. Trace:
I20240426 09:55:29.826275 10016 raft_consensus.cc:2379] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:29.827566 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:29.846978 11807 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.849227 11807 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:29.826628 9938 rpcz_store.cc:276] 0426 09:55:27.489552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.167825 (+678273us) service_pool.cc:225] Handling call
0426 09:55:28.168096 (+ 271us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:28.196242 (+ 28146us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:28.196272 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:28.196698 (+ 426us) log.cc:842] Serialized 26 byte log entry
0426 09:55:28.365187 (+168489us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:28.365225 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:28.365291 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:29.815153 (+1449862us) spinlock_profiling.cc:243] Waited 4.96 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:55:29.825830 (+ 10677us) raft_consensus.cc:1671] finished
0426 09:55:29.825847 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:29.826104 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27835,"raft.queue_time_us":30455,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":14861,"spinlock_wait_cycles":11415808,"thread_start_us":101064,"threads_started":1,"wal-append.queue_time_us":279046}
I20240426 09:55:29.852373 11504 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:29.881318 11735 raft_consensus.cc:2781] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:29.892596 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1032) took 1493 ms. Trace:
I20240426 09:55:29.895713 10010 rpcz_store.cc:276] 0426 09:55:28.399129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.455335 (+ 56206us) service_pool.cc:225] Handling call
0426 09:55:28.455567 (+ 232us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:28.857285 (+401718us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.889880 (+1032595us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b54000564d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:29.892541 (+ 2661us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":921529216}
I20240426 09:55:29.891923 11473 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:29.900697 9979 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:29.847667 11809 consensus_queue.cc:1046] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20240426 09:55:29.836323 11621 consensus_queue.cc:260] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:29.895673 11735 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:55:29.909350 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 94769.4us
I20240426 09:55:29.911106 10012 raft_consensus.cc:1261] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20240426 09:55:29.911454 10012 pending_rounds.cc:76] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting all ops after (but not including) 1
I20240426 09:55:29.911653 10012 pending_rounds.cc:97] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240426 09:55:29.911837 10012 raft_consensus.cc:2866] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:29.906345 11805 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:29.911037 11735 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:29.927052 11502 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:29.929811 11502 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.932814 11782 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:29.920712 11772 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:29.922448 11621 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:29.945011 11693 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:29.945574 10010 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:29.945693 11693 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:29.946822 10010 raft_consensus.cc:595] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:29.947178 10010 raft_consensus.cc:603] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 09:55:29.947837 11693 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:29.948724 11502 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:29.949155 9939 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
W20240426 09:55:29.949290 11733 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.261s user 0.016s sys 0.016s
W20240426 09:55:29.949761 11733 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.294s user 0.016s sys 0.016s
I20240426 09:55:29.950958 11740 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:29.955825 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240426 09:55:29.962548 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:55:29.944617 11787 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.007s sys 0.007s
I20240426 09:55:29.997277 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:55:30.002799 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:30.004045 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.004638 10090 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:30.005420 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.005991 10090 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:30.008000 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.008338 11646 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:29.995748 11502 consensus_queue.cc:1035] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20240426 09:55:29.960613 11772 raft_consensus.cc:686] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:30.006757 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.015888 10087 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:30.017215 10086 raft_consensus.cc:2379] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 4: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:29.920783 11805 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:29.991803 11787 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.614s user 0.007s sys 0.008s
I20240426 09:55:30.031631 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:30.030639 10090 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:55:30.016649 11729 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:30.040032 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.050226 10611 raft_consensus.cc:1226] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:30.051453 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.054268 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:30.032887 11774 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.065117 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.040769 11740 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.032542 11733 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.067601 11782 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:30.071378 11697 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:30.071825 11771 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:30.072510 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 129536us
I20240426 09:55:30.065076 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:30.079237 10090 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:30.064635 11688 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.088518 10056 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:30.088954 11688 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:30.089458 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 49981.1us
I20240426 09:55:30.090878 11812 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:30.091053 11688 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:30.091588 11688 raft_consensus.cc:2726] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:30.091436 11812 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:30.092145 11688 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:30.092617 11688 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.093116 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:30.067843 11729 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.094964 11778 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.094594 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.092520 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.095623 11536 consensus_queue.cc:1046] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:30.091202 9940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1087) took 2082 ms. Trace:
I20240426 09:55:30.096536 11582 raft_consensus.cc:3032] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:30.097021 11582 raft_consensus.cc:729] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:30.097338 11772 consensus_queue.cc:237] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.097822 11454 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:30.099090 11454 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.099224 11782 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.096472 9940 rpcz_store.cc:276] 0426 09:55:28.009109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.853016 (+843907us) service_pool.cc:225] Handling call
0426 09:55:29.767071 (+914055us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:29.767672 (+ 601us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:30.091035 (+323363us) spinlock_profiling.cc:243] Waited 913 ms on lock 0x7b54003769d0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c45b99 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:30.091155 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2101005696}
I20240426 09:55:30.071264 11680 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.102738 10056 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:30.104770 10087 raft_consensus.cc:1226] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240426 09:55:30.072444 11771 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.078095 11783 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.087082 11230 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.107065 11646 consensus_queue.cc:1046] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20240426 09:55:30.112232 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:30.096753 10082 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:30.113149 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.070159 11774 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.096598 11536 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.115870 11454 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:30.136547 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.137192 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.140044 11230 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.142148 11230 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.097851 11582 consensus_queue.cc:260] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.120790 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.145056 9935 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:30.145323 11582 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:30.121528 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1023) took 1726 ms. Trace:
I20240426 09:55:30.145728 10011 rpcz_store.cc:276] 0426 09:55:28.394746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.396300 (+ 1554us) service_pool.cc:225] Handling call
0426 09:55:28.396593 (+ 293us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:28.396787 (+ 194us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:28.396811 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:28.397198 (+ 387us) log.cc:842] Serialized 26 byte log entry
0426 09:55:29.703775 (+1306577us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:29.703840 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:29.703909 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:30.114968 (+411059us) spinlock_profiling.cc:243] Waited 9.47 ms on lock 0x7b540013bb20. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2bd48f1 00007f9ca2bdf3b0 00007f9ca2bdf358 00007f9ca2bdf2d8 00007f9ca2bdf293 00007f9ca2bde51c 00007f9ca8c4ee26 00007f9ca8c47428 00007f9ca2c1b51b 00007f9ca2c1e010
0426 09:55:30.121252 (+ 6284us) raft_consensus.cc:1671] finished
0426 09:55:30.121273 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:30.121466 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21788544,"thread_start_us":1229165,"threads_started":1,"wal-append.queue_time_us":1174387}
W20240426 09:55:30.146910 11818 log.cc:924] Time spent T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20240426 09:55:30.117703 11688 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:30.108947 11733 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:30.101035 11808 log.cc:924] Time spent T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.185s user 0.001s sys 0.000s
I20240426 09:55:30.124526 11778 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.095224 11729 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:30.130453 11462 raft_consensus.cc:2726] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:30.130649 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:30.130110 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.144503 11812 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:30.130086 9937 raft_consensus.cc:1226] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240426 09:55:30.148636 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:30.155542 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:30.071935 11697 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.154618 10090 raft_consensus.cc:2454] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:30.146087 11807 raft_consensus.cc:3032] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:30.168293 9906 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:30.147380 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:30.174119 11778 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.149067 11783 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.176911 11783 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.145843 11771 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:30.173534 10082 raft_consensus.cc:1226] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:30.168257 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:30.182646 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 232398us
I20240426 09:55:30.123939 11680 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.183738 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:30.183764 10055 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:30.172055 11807 raft_consensus.cc:729] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:30.194974 9936 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:30.186606 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.201737 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.204856 10090 raft_consensus.cc:1226] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 09:55:30.205013 11778 consensus_queue.cc:1046] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:30.205245 11688 consensus_queue.cc:1035] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:30.194689 11812 raft_consensus.cc:2726] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:30.198872 11782 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.207512 11774 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:30.209214 11513 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:30.203186 10087 raft_consensus.cc:1261] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:55:30.214026 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.215682 10090 raft_consensus.cc:1226] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:30.216065 11697 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:30.216816 10090 raft_consensus.cc:1226] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:30.219357 9933 raft_consensus.cc:1226] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6. Ops: [4.2-5.3]
I20240426 09:55:30.220244 11408 raft_consensus.cc:3032] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:30.220605 11408 raft_consensus.cc:729] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:30.221455 11408 consensus_queue.cc:260] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.222254 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:30.181247 11536 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.222680 11408 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:30.246220 11536 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.228039 10082 raft_consensus.cc:1261] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240426 09:55:30.198323 11771 raft_consensus.cc:2781] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:30.248183 11771 raft_consensus.cc:686] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:30.249130 11771 consensus_queue.cc:237] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.215595 10056 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:30.251893 11474 raft_consensus.cc:3032] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Stepping down as leader of term 3
W20240426 09:55:30.232462 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 106297us
I20240426 09:55:30.253414 10056 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:30.253863 9906 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:30.254189 11774 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:30.254743 11774 raft_consensus.cc:686] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:30.255416 11812 consensus_queue.cc:1035] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:30.255649 11774 consensus_queue.cc:237] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.234274 11405 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:55:30.240770 10013 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1333) took 2304 ms (client timeout 1652 ms). Trace:
W20240426 09:55:30.256765 10013 rpcz_store.cc:269] 0426 09:55:27.935922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:27.937349 (+ 1427us) service_pool.cc:225] Handling call
0426 09:55:30.240432 (+2303083us) spinlock_profiling.cc:243] Waited 2 s on lock 0x7b5400056ed4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:30.240574 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4601682304}
I20240426 09:55:30.252331 11474 raft_consensus.cc:729] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:30.199281 11733 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.259824 11474 consensus_queue.cc:260] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.261140 11474 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:30.239547 10015 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:30.200897 11805 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.264755 11782 raft_consensus.cc:2726] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:55:30.267035 10618 raft_consensus.cc:2454] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:30.270735 10056 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:30.271292 10090 raft_consensus.cc:1261] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:30.272150 10056 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:30.293321 9936 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:30.295929 10015 raft_consensus.cc:2454] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:30.297596 9979 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:30.299221 10583 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:30.300549 10583 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:30.303290 9905 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:30.313899 9979 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:30.322234 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:30.322944 11680 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:30.324546 9932 raft_consensus.cc:1226] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:30.326016 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.329568 11735 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.331099 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.332816 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:30.333277 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:30.329978 11680 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:30.333990 9906 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:30.334491 9979 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:30.333415 10015 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:30.350561 9940 raft_consensus.cc:1226] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:30.351935 9938 raft_consensus.cc:1226] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:30.354524 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.223174 10083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 953) took 2531 ms. Trace:
I20240426 09:55:30.220935 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240426 09:55:30.353858 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 50651.5us
I20240426 09:55:30.296164 11697 raft_consensus.cc:2726] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:30.358485 9979 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:30.313460 11623 raft_consensus.cc:2726] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:30.313333 11694 raft_consensus.cc:2781] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:30.369706 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:30.303671 11536 raft_consensus.cc:2781] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:30.354056 11582 raft_consensus.cc:2781] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:30.378734 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:30.374091 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.375042 9906 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:30.372982 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:30.380479 10015 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:30.400534 10086 raft_consensus.cc:2454] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:30.360181 11504 raft_consensus.cc:2781] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:30.226958 11807 consensus_queue.cc:260] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:30.349196 11735 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.402292 10012 raft_consensus.cc:2454] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:30.416524 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.405409 11582 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.355600 10082 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:30.432632 10015 raft_consensus.cc:1261] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:30.424306 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.418172 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:30.429476 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:30.373020 11783 raft_consensus.cc:2726] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:55:30.430495 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:30.431254 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 55702.6us
I20240426 09:55:30.435423 9979 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:30.374441 11809 raft_consensus.cc:2781] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:30.378913 11408 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:30.361212 11680 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.379796 11536 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:30.414848 11504 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:30.381317 11694 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:30.442360 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.444766 11807 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:30.469439 10089 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 975) took 1065 ms. Trace:
I20240426 09:55:30.470724 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.469810 10089 rpcz_store.cc:276] 0426 09:55:29.404156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:29.706376 (+302220us) service_pool.cc:225] Handling call
0426 09:55:29.706720 (+ 344us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:30.105541 (+398821us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:30.105570 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:30.148679 (+ 43109us) log.cc:842] Serialized 26 byte log entry
0426 09:55:30.344793 (+196114us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:55:30.415108 (+ 70315us) log.cc:842] Serialized 14 byte log entry
0426 09:55:30.415313 (+ 205us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:30.415383 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:30.469258 (+ 53875us) raft_consensus.cc:1671] finished
0426 09:55:30.469276 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:30.469387 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":339650,"prepare.queue_time_us":118681,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":368339,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":14092,"thread_start_us":258806,"threads_started":2,"wal-append.queue_time_us":209985}
I20240426 09:55:30.433560 11778 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:30.472165 11778 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.445149 11829 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.473919 11829 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.445616 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
W20240426 09:55:30.446215 9887 auto_rebalancer.cc:663] Could not move replica: Not found: Couldn't find leader for tablet 6151686b1a24447880e49b7d7395fcc1
I20240426 09:55:30.414777 11688 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.446794 11772 consensus_queue.cc:1035] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 09:55:30.447738 11408 raft_consensus.cc:686] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:30.487398 11408 consensus_queue.cc:237] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.447899 10055 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:30.448710 11230 consensus_queue.cc:1046] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20240426 09:55:30.450657 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 75709.6us
I20240426 09:55:30.458971 11536 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:30.491577 9906 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:30.418574 11723 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:30.492393 11723 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:30.494436 11723 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:30.454746 11729 raft_consensus.cc:2726] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:30.456120 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:30.496264 10014 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:30.432282 11582 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:30.462865 11694 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:30.403802 11462 consensus_queue.cc:1046] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20240426 09:55:30.432572 11814 raft_consensus.cc:2726] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:30.466496 9939 raft_consensus.cc:1261] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:55:30.452946 11733 raft_consensus.cc:2781] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:30.466331 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.467608 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.468392 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.467212 9938 raft_consensus.cc:1226] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240426 09:55:30.466389 9836 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50254 (request call id 191) took 5646 ms. Trace:
I20240426 09:55:30.506362 9836 rpcz_store.cc:276] 0426 09:55:24.819945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:24.820077 (+ 132us) service_pool.cc:225] Handling call
0426 09:55:25.195679 (+375602us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54000041d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcbc70 00007f9ca2c4f0ba 00007f9ca8ba096c 00007f9ca8c9a8de 00007f9ca6faea04 00007f9ca6fae992 00007f9ca6fae8c1 00007f9ca6fae83c 00007f9ca6fadaf2 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:30.466175 (+5270496us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:28.795057 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:28.796298 (+ 1241us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:28.796331 (+ 33us) write_op.cc:427] Acquired schema lock
0426 09:55:28.796348 (+ 17us) tablet.cc:660] Decoding operations
0426 09:55:28.797100 (+ 752us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:28.797147 (+ 47us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:28.797175 (+ 28us) tablet.cc:683] Acquiring locks for 6 operations
0426 09:55:28.797645 (+ 470us) tablet.cc:699] Row locks acquired
0426 09:55:28.797660 (+ 15us) write_op.cc:260] PREPARE: finished
0426 09:55:28.797796 (+ 136us) write_op.cc:270] Start()
0426 09:55:28.797899 (+ 103us) write_op.cc:276] Timestamp: P: 1714125328797751 usec, L: 0
0426 09:55:28.797915 (+ 16us) op_driver.cc:331] REPLICATION: starting
0426 09:55:28.798610 (+ 695us) log.cc:842] Serialized 2563 byte log entry
0426 09:55:30.074962 (+1276352us) op_driver.cc:444] REPLICATION: finished
0426 09:55:30.115409 (+ 40447us) write_op.cc:301] APPLY: starting
0426 09:55:30.115542 (+ 133us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:30.115732 (+ 190us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:30.115826 (+ 94us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:30.116569 (+ 743us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:30.116741 (+ 172us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0426 09:55:30.116781 (+ 40us) write_op.cc:312] APPLY: finished
0426 09:55:30.117000 (+ 219us) log.cc:842] Serialized 52 byte log entry
0426 09:55:30.117139 (+ 139us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:30.117270 (+ 131us) write_op.cc:446] Released schema lock
0426 09:55:30.117311 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0426 09:55:30.117550 (+ 239us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:30.117577 (+ 27us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":1151226,"reported_tablets":12,"spinlock_wait_cycles":256030464,"child_traces":[["op",{"apply.queue_time_us":17543,"apply.run_cpu_time_us":2476,"apply.run_wall_time_us":2476,"num_ops":6,"prepare.queue_time_us":512576,"prepare.run_cpu_time_us":3644,"prepare.run_wall_time_us":3770,"replication_time_us":1267194}]]}
W20240426 09:55:30.478449 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 102018us
I20240426 09:55:30.481884 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.511319 10085 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:30.483150 11830 raft_consensus.cc:2726] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:30.483736 9839 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58806 (request call id 85) took 9537 ms. Trace:
I20240426 09:55:30.486099 11783 consensus_queue.cc:1035] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:30.488075 9979 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:30.519125 11644 consensus_queue.cc:1046] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.162s
I20240426 09:55:30.477156 11504 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
W20240426 09:55:30.519883 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 72080.3us
I20240426 09:55:30.495581 10088 raft_consensus.cc:1261] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 09:55:30.440482 11454 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:30.521075 11454 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.521584 11680 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:30.521869 11771 raft_consensus.cc:3032] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:30.522236 11771 raft_consensus.cc:729] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:30.448364 11739 consensus_queue.cc:1046] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 09:55:30.440436 11809 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 09:55:30.531919 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 43967.4us
I20240426 09:55:30.501046 9934 raft_consensus.cc:1261] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:30.533496 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.475908 11829 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:30.466897 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:30.477102 9887 auto_rebalancer.cc:264] scheduled replica move failed to complete: Not found: Couldn't find leader for tablet 6151686b1a24447880e49b7d7395fcc1
I20240426 09:55:30.474033 11778 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.516981 9906 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:30.520792 10055 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:30.523012 11771 consensus_queue.cc:260] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.538370 11809 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:30.547559 11771 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:30.515928 9839 rpcz_store.cc:276] 0426 09:55:20.946166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:20.946307 (+ 141us) service_pool.cc:225] Handling call
0426 09:55:20.964799 (+ 18492us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b54000041d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcbc70 00007f9ca2c4f0ba 00007f9ca8ba096c 00007f9ca8c9a8de 00007f9ca6faea04 00007f9ca6fae992 00007f9ca6fae8c1 00007f9ca6fae83c 00007f9ca6fadaf2 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:30.483542 (+9518743us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:28.248834 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:28.249391 (+ 557us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:28.249429 (+ 38us) write_op.cc:427] Acquired schema lock
0426 09:55:28.249449 (+ 20us) tablet.cc:660] Decoding operations
0426 09:55:28.276443 (+ 26994us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:28.290187 (+ 13744us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:28.300123 (+ 9936us) tablet.cc:683] Acquiring locks for 3 operations
0426 09:55:28.401238 (+101115us) tablet.cc:699] Row locks acquired
0426 09:55:28.401721 (+ 483us) write_op.cc:260] PREPARE: finished
0426 09:55:28.429775 (+ 28054us) write_op.cc:270] Start()
0426 09:55:28.439991 (+ 10216us) write_op.cc:276] Timestamp: P: 1714125328429728 usec, L: 0
0426 09:55:28.444898 (+ 4907us) op_driver.cc:331] REPLICATION: starting
0426 09:55:28.553029 (+108131us) log.cc:842] Serialized 1301 byte log entry
0426 09:55:29.096088 (+543059us) op_driver.cc:444] REPLICATION: finished
0426 09:55:29.819341 (+723253us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9c9fcfde2e 00007f9ca8b37789 00007f9c9fd1b0eb 00007f9c9fd194ef 00007f9c9fd1b652 00007f9ca6657ac0
0426 09:55:30.008797 (+189456us) write_op.cc:301] APPLY: starting
0426 09:55:30.008950 (+ 153us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:30.009144 (+ 194us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:30.009182 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:30.009773 (+ 591us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:30.010006 (+ 233us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0426 09:55:30.010061 (+ 55us) write_op.cc:312] APPLY: finished
0426 09:55:30.010415 (+ 354us) log.cc:842] Serialized 34 byte log entry
0426 09:55:30.010629 (+ 214us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:30.010802 (+ 173us) write_op.cc:446] Released schema lock
0426 09:55:30.010871 (+ 69us) write_op.cc:341] FINISH: Updating metrics
0426 09:55:30.011214 (+ 343us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:30.011249 (+ 35us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":2195199,"reported_tablets":3,"spinlock_wait_cycles":25888640,"child_traces":[["op",{"apply.queue_time_us":629151,"apply.run_cpu_time_us":2811,"apply.run_wall_time_us":49800,"num_ops":3,"prepare.queue_time_us":67221,"prepare.run_cpu_time_us":40676,"prepare.run_wall_time_us":526297,"raft.queue_time_us":1190787,"raft.run_cpu_time_us":19435,"raft.run_wall_time_us":1069194,"replication_time_us":614311,"spinlock_wait_cycles":55239040,"thread_start_us":1130361,"threads_started":4,"wal-append.queue_time_us":233946}]]}
I20240426 09:55:30.512712 11733 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:30.548967 11733 raft_consensus.cc:3037] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:30.531047 11454 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:30.521530 11769 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:30.550562 11769 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.538568 10081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 971) took 1197 ms. Trace:
I20240426 09:55:30.551282 11769 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:30.555902 11644 raft_consensus.cc:3032] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:30.556303 11644 raft_consensus.cc:729] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:30.557092 11644 consensus_queue.cc:260] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.558300 11644 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:30.558868 11739 raft_consensus.cc:3032] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:30.559247 11739 raft_consensus.cc:729] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:30.560160 11739 consensus_queue.cc:260] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.561365 11739 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:30.562444 11733 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:30.534086 11836 log.cc:924] Time spent T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240426 09:55:30.539220 11772 raft_consensus.cc:2781] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:30.563880 11772 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.503767 11688 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.564263 11772 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:30.565702 11688 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:30.566277 10007 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:30.566743 11230 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:30.567173 11230 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.544137 11721 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.568668 11721 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.564257 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:30.570748 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 50018.4us
I20240426 09:55:30.563059 11804 raft_consensus.cc:2726] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:30.544538 11693 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:30.572096 11693 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.540805 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:30.574203 10611 raft_consensus.cc:2454] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:30.502139 11757 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:30.545197 11536 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:30.575245 11757 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.576455 10013 raft_consensus.cc:1226] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:30.578914 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.540030 9838 catalog_manager.cc:5484] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 1 to 2, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 2 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:30.370322 10083 rpcz_store.cc:276] 0426 09:55:27.691980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:28.158351 (+466371us) service_pool.cc:225] Handling call
0426 09:55:28.159059 (+ 708us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:28.195722 (+ 36663us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:28.197936 (+ 2214us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:28.278386 (+ 80450us) log.cc:842] Serialized 26 byte log entry
0426 09:55:28.547704 (+269318us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:28.642677 (+ 94973us) log.cc:842] Serialized 14 byte log entry
0426 09:55:28.732178 (+ 89501us) spinlock_profiling.cc:243] Waited 2.17 ms on lock 0x7b18001f17a0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fd1bfa4 00007f9c9fd1f351 00007f9c9fd1f309 00007f9c9fd1f299 00007f9c9fd1f261 00007f9c9fd1e55d 00007f9ca8f8fd34 00007f9ca8f8fb69 00007f9c9fcfe5f6
0426 09:55:28.869180 (+137002us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:28.869347 (+ 167us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:29.545906 (+676559us) spinlock_profiling.cc:243] Waited 4.15 ms on lock 0x7b18001f17a0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fce97c6 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fd1be78 00007f9c9fd1f351
0426 09:55:30.209621 (+663715us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b18001f17a0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fd1be78 00007f9c9fd1f351 00007f9c9fd1f309 00007f9c9fd1f299 00007f9c9fd1f261 00007f9c9fd1e55d 00007f9ca8f8fd34 00007f9ca8f8fb69 00007f9c9fcfe5f6
0426 09:55:30.222851 (+ 13230us) raft_consensus.cc:1671] finished
0426 09:55:30.222871 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:30.223010 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":161,"prepare.queue_time_us":51360,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":61877,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":6161,"spinlock_wait_cycles":19576960,"thread_start_us":350430,"threads_started":2,"wal-append.queue_time_us":353121}
I20240426 09:55:30.588793 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:30.589360 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 72490.2us
I20240426 09:55:30.589908 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.590368 11809 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.591874 11805 raft_consensus.cc:2726] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:30.592461 11809 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.592957 11805 consensus_queue.cc:1046] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:30.571426 10055 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:30.590535 9933 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:30.596112 10055 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:30.600411 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:30.602098 10014 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:30.602362 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.585772 9940 raft_consensus.cc:2379] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:30.585786 11729 raft_consensus.cc:2781] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:30.551477 10081 rpcz_store.cc:276] 0426 09:55:29.341181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:29.686418 (+345237us) service_pool.cc:225] Handling call
0426 09:55:29.720209 (+ 33791us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 09:55:30.041873 (+321664us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:30.041898 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 09:55:30.042396 (+ 498us) log.cc:842] Serialized 52 byte log entry
0426 09:55:30.062065 (+ 19669us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:55:30.254033 (+191968us) spinlock_profiling.cc:243] Waited 7.15 ms on lock 0x7b18000bd9a0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fd1be78 00007f9c9fd1f351 00007f9c9fd1f309 00007f9c9fd1f299 00007f9c9fd1f261 00007f9c9fd1e55d 00007f9ca8f8fd34 00007f9ca8f8fb69 00007f9c9fcfe5f6
0426 09:55:30.373412 (+119379us) log.cc:842] Serialized 14 byte log entry
0426 09:55:30.505867 (+132455us) log.cc:842] Serialized 14 byte log entry
0426 09:55:30.531654 (+ 25787us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:30.531722 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:30.538384 (+ 6662us) raft_consensus.cc:1671] finished
0426 09:55:30.538403 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:30.538522 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":380959,"prepare.queue_time_us":99321,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":64,"raft.queue_time_us":385422,"raft.run_cpu_time_us":443,"raft.run_wall_time_us":55143,"spinlock_wait_cycles":16454912,"thread_start_us":383594,"threads_started":4,"wal-append.queue_time_us":81121}
I20240426 09:55:30.570390 11741 raft_consensus.cc:3032] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:30.604521 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:30.610240 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.610656 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.610846 9934 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:30.611032 9979 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:30.612929 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:30.613484 9906 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:30.619904 10583 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:30.621001 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.622305 9906 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:30.622861 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.622437 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:30.624728 10014 raft_consensus.cc:2396] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240426 09:55:30.625315 10055 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:30.621757 9937 raft_consensus.cc:2454] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:30.627099 10056 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:30.602612 11721 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.601446 11536 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:30.634747 10089 raft_consensus.cc:1226] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:30.607606 11729 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:30.608965 11679 raft_consensus.cc:2781] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:30.633987 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.606812 11829 raft_consensus.cc:2781] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:30.638247 9979 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:30.639360 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:30.606670 11741 raft_consensus.cc:729] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:30.635747 9838 catalog_manager.cc:5484] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 1 to 2, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 2 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:30.616731 11778 raft_consensus.cc:2781] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:30.619547 11473 raft_consensus.cc:3032] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:30.650087 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.652478 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.631748 11807 raft_consensus.cc:2781] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:30.645633 11829 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:30.663149 11829 raft_consensus.cc:3037] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:30.633636 11812 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:30.664099 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.673246 9935 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:30.669693 9838 catalog_manager.cc:5484] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 2 to 3, leader changed from 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 3 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:30.663458 11582 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.680186 9980 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:30.670814 9982 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:30.609462 11828 consensus_queue.cc:1035] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20240426 09:55:30.642582 11679 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.645206 11697 raft_consensus.cc:2781] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:30.685824 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:30.703913 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:30.710791 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 128359us
I20240426 09:55:30.617094 11694 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:30.645057 11623 raft_consensus.cc:2726] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:30.721483 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.644943 11825 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:30.649811 11733 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.650429 11778 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.630458 11789 consensus_queue.cc:1046] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
W20240426 09:55:30.741355 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 123523us
I20240426 09:55:30.765802 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:30.739972 11783 raft_consensus.cc:2781] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:30.790496 11772 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.791044 11783 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:30.791450 11783 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:30.792485 11772 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.792976 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.793324 10015 raft_consensus.cc:2379] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:30.743434 9982 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:30.657312 11473 raft_consensus.cc:729] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:30.748790 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:30.670315 11830 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:30.747026 9838 catalog_manager.cc:5484] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 1 to 2, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 2 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:30.677829 11812 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:30.670202 11723 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:30.677037 11807 raft_consensus.cc:686] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:30.724048 11679 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:30.723352 11697 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:30.642217 11729 raft_consensus.cc:3037] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:30.766919 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:30.742547 11721 raft_consensus.cc:2726] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:30.768400 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:30.831876 10009 raft_consensus.cc:2454] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:30.769827 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.770906 11693 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.807109 11545 raft_consensus.cc:2726] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:55:30.772282 11694 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.717521 11533 consensus_queue.cc:1035] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
W20240426 09:55:30.775728 11851 log.cc:924] Time spent T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20240426 09:55:30.775978 10056 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
W20240426 09:55:30.836460 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 60562.4us
I20240426 09:55:30.837316 11828 raft_consensus.cc:2726] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:30.776271 10616 raft_consensus.cc:1226] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:30.703859 11810 consensus_queue.cc:1035] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.150s
I20240426 09:55:30.839417 10014 raft_consensus.cc:1261] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:30.681898 11846 consensus_queue.cc:1035] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.111s
I20240426 09:55:30.708029 11741 consensus_queue.cc:260] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.842003 11741 raft_consensus.cc:3037] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:30.784539 11230 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.785867 11582 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:30.847286 11536 consensus_queue.cc:1159] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d3a480e831d649139ff016fb68aee3ee" responder_term: 3 status { last_received { term: 2 index: 2 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 09:55:30.767289 11778 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:30.789968 11789 raft_consensus.cc:3032] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:30.850960 11789 raft_consensus.cc:729] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:30.851140 11697 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:30.765282 11825 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:30.853277 11825 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:30.790948 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:30.746342 10010 raft_consensus.cc:2379] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:30.794473 11504 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:30.855945 11504 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:30.859398 11807 consensus_queue.cc:237] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.794446 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:30.800818 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.862617 11473 consensus_queue.cc:260] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.863696 11473 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:30.864089 10090 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:55:30.782505 11757 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:30.816015 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 72655.7us
I20240426 09:55:30.820788 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:30.815869 11812 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:30.824460 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:30.782618 11769 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.722994 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.867668 10089 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:30.824187 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:30.870519 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 44607.9us
I20240426 09:55:30.873693 9982 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:30.809278 11830 raft_consensus.cc:686] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:30.876297 9838 catalog_manager.cc:5484] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 1 to 2, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 2 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:30.881232 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:30.881830 10007 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:30.882251 11769 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:30.882934 10083 raft_consensus.cc:1226] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 5. Current term is 6. Ops: []
I20240426 09:55:30.884469 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:30.887071 11778 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.870314 11804 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:30.887960 11804 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.889819 11804 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:30.816407 11723 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.890676 11723 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:30.868862 10056 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:30.893728 9982 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:30.892984 11230 raft_consensus.cc:2726] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:30.900517 9982 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:30.895643 9904 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:30.910367 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:30.911218 10015 raft_consensus.cc:2379] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:30.894899 9838 catalog_manager.cc:5484] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 3 to 4, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 4 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:30.912398 9904 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:30.919173 10012 raft_consensus.cc:1226] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:30.921965 9904 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:30.923086 9982 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:30.937285 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 68506.8us
I20240426 09:55:30.939199 10056 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:30.940171 9838 catalog_manager.cc:5484] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 1 to 2, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 2 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:30.913117 11769 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:30.902403 11741 raft_consensus.cc:2726] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:30.948693 11513 raft_consensus.cc:2726] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:30.959864 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:30.938282 11757 raft_consensus.cc:2726] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:55:30.894601 11504 consensus_queue.cc:1046] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20240426 09:55:30.942836 11545 raft_consensus.cc:2726] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:30.948117 11688 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:30.969272 10583 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:30.949224 11230 raft_consensus.cc:2781] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:30.970285 9982 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:30.950868 11696 raft_consensus.cc:2726] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:30.978752 11628 raft_consensus.cc:2726] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:30.981495 9904 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:30.982419 9904 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:30.988903 10082 raft_consensus.cc:1261] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:30.891347 11830 consensus_queue.cc:237] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:30.996011 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.990017 9982 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:30.990857 10056 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:30.995069 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:30.957760 11805 consensus_queue.cc:1035] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20240426 09:55:30.999804 10056 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:30.986495 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:31.002069 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:31.005236 10056 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:31.002816 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:31.005837 9937 raft_consensus.cc:3037] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:31.007755 9935 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:55:30.974449 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:31.007835 9936 pending_rounds.cc:76] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9: Aborting all ops after (but not including) 1
I20240426 09:55:30.963364 11769 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.013559 9936 pending_rounds.cc:97] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240426 09:55:31.008877 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:30.980981 11545 raft_consensus.cc:2726] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:31.020982 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:30.976893 11757 raft_consensus.cc:3032] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:31.022043 9936 raft_consensus.cc:2866] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:31.025813 10082 raft_consensus.cc:2379] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:30.997440 11782 raft_consensus.cc:2726] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:30.998915 11772 raft_consensus.cc:2726] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:31.001977 11771 raft_consensus.cc:2726] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:31.013680 11454 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:31.038033 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:30.990447 11729 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.051980 11462 raft_consensus.cc:2781] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240426 09:55:31.063191 11644 consensus_meta.cc:287] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.505s user 0.009s sys 0.000s
I20240426 09:55:31.064031 11858 raft_consensus.cc:2726] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:31.064765 11679 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.026487 11810 raft_consensus.cc:2741] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Leader pre-election decision vote started in defunct term 4: won
I20240426 09:55:31.075374 9941 raft_consensus.cc:1261] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240426 09:55:31.075724 9941 pending_rounds.cc:76] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9: Aborting all ops after (but not including) 2
I20240426 09:55:31.075909 9941 pending_rounds.cc:97] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240426 09:55:31.076135 9941 raft_consensus.cc:2866] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:31.000166 11230 raft_consensus.cc:686] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:31.009063 11513 consensus_queue.cc:1035] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.044s
I20240426 09:55:31.060658 11789 consensus_queue.cc:260] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:31.068420 10008 raft_consensus.cc:1261] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20240426 09:55:31.069628 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:31.091950 10011 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:31.013163 11582 consensus_queue.cc:1046] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.044s
I20240426 09:55:31.070960 11778 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:31.025604 11757 raft_consensus.cc:729] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
W20240426 09:55:31.080535 11859 log.cc:924] Time spent T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.137s user 0.003s sys 0.000s
I20240426 09:55:31.082041 11697 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.102015 11697 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:31.033659 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:31.112613 10012 raft_consensus.cc:1226] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:31.082607 11536 consensus_queue.cc:1046] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240426 09:55:31.083333 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 78170.1us
I20240426 09:55:31.084965 11454 raft_consensus.cc:2726] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:31.086199 10089 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:31.080578 9982 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:31.093336 11504 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:31.089524 10087 raft_consensus.cc:2379] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:31.091594 10008 pending_rounds.cc:76] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting all ops after (but not including) 2
I20240426 09:55:31.118459 10008 pending_rounds.cc:97] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20240426 09:55:31.093519 10090 raft_consensus.cc:1226] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:31.063674 11863 consensus_queue.cc:1046] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 09:55:31.074273 10611 raft_consensus.cc:2454] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:31.078491 9937 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:31.121169 10015 raft_consensus.cc:1226] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240426 09:55:31.116127 10056 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:31.118716 10008 raft_consensus.cc:2866] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:31.122189 10082 raft_consensus.cc:1226] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:31.122303 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:31.123822 10013 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:31.123772 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:31.125233 10012 raft_consensus.cc:2379] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:31.074304 11812 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.074873 11829 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.133697 9982 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:31.115010 11533 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:31.129087 9904 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:31.124516 10084 raft_consensus.cc:2379] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 5: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:31.102720 9932 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:31.096921 11858 consensus_queue.cc:1046] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20240426 09:55:31.143903 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:31.146795 9905 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:31.149125 11805 raft_consensus.cc:2781] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:31.117983 11729 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:31.151511 10081 raft_consensus.cc:2354] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5.
I20240426 09:55:31.131553 11828 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:31.120720 11504 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.120976 11846 consensus_queue.cc:1046] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20240426 09:55:31.156538 10014 raft_consensus.cc:1226] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:31.132905 11502 raft_consensus.cc:2726] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:31.119199 11783 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:31.135763 11680 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:31.145552 11782 raft_consensus.cc:2781] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:31.149718 11739 raft_consensus.cc:2726] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:31.167582 11680 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:31.168018 11680 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:31.154070 11454 consensus_queue.cc:1035] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20240426 09:55:31.168632 11783 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:31.152465 11805 raft_consensus.cc:686] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:31.164400 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:31.142941 11536 consensus_queue.cc:1035] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.024s
I20240426 09:55:31.152909 11679 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:31.163602 9982 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:31.148483 11533 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.177649 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:31.178259 9982 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:31.157021 11828 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.180711 10014 raft_consensus.cc:2379] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:31.154418 11697 consensus_queue.cc:1035] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20240426 09:55:31.167905 11782 raft_consensus.cc:686] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:31.165187 11545 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.015s
W20240426 09:55:31.183082 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 55506.5us
I20240426 09:55:31.185678 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:31.185609 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:31.190620 9937 raft_consensus.cc:2379] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 5: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:31.189384 10056 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:31.189512 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:31.192557 10010 raft_consensus.cc:1261] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20240426 09:55:31.192895 10010 pending_rounds.cc:76] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting all ops after (but not including) 2
I20240426 09:55:31.192955 10056 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:31.193475 10010 pending_rounds.cc:97] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20240426 09:55:31.193960 10010 raft_consensus.cc:2866] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:31.192935 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:31.197283 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:31.197779 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:31.198276 11679 consensus_queue.cc:1035] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:31.198710 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:31.192492 11693 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:31.200385 11679 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:31.200246 11693 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.201128 11825 raft_consensus.cc:2726] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:31.202735 11533 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:31.203671 11462 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:31.203783 9935 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:31.204114 11462 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:31.156163 11757 consensus_queue.cc:260] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.208667 11757 raft_consensus.cc:3037] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:31.180794 11778 raft_consensus.cc:2781] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:31.209573 11778 raft_consensus.cc:686] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:31.209935 11789 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:31.210433 11778 consensus_queue.cc:237] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.212888 9935 raft_consensus.cc:1261] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:31.195024 11691 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:31.202273 10012 raft_consensus.cc:1226] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:31.218521 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:31.219005 11408 raft_consensus.cc:3032] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:31.219280 9937 raft_consensus.cc:2379] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:31.202354 9938 raft_consensus.cc:1261] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240426 09:55:31.220551 11828 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:31.200860 11679 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.220992 11582 raft_consensus.cc:3032] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:31.222510 11782 consensus_queue.cc:237] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.222617 11679 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:31.222965 11582 raft_consensus.cc:729] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:31.224665 11582 consensus_queue.cc:260] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.225782 11408 raft_consensus.cc:729] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:31.229215 11582 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:31.229667 11408 consensus_queue.cc:260] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:31.197221 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:31.226192 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:31.227209 10014 raft_consensus.cc:2354] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4.
I20240426 09:55:31.228159 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:31.234759 11536 consensus_queue.cc:1035] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:31.235014 11810 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:31.236162 10056 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:31.236353 11810 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.238440 11771 consensus_queue.cc:1035] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:31.184448 11769 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:31.250762 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:31.276549 9904 leader_election.cc:398] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5.
I20240426 09:55:31.277004 9904 leader_election.cc:401] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20240426 09:55:31.237430 11828 raft_consensus.cc:2726] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:31.201139 11809 raft_consensus.cc:3032] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:31.177613 11741 consensus_queue.cc:1035] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20240426 09:55:31.275172 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:31.277539 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 49794.4us
I20240426 09:55:31.161660 11230 consensus_queue.cc:237] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.198364 9932 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 5
I20240426 09:55:31.315852 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:31.239439 11739 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:31.324890 10011 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:31.203773 11693 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:31.239465 11408 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:31.355715 10056 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:31.435983 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:31.373167 9934 raft_consensus.cc:1261] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
W20240426 09:55:31.491822 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 136175us
I20240426 09:55:31.492291 9934 pending_rounds.cc:76] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9: Aborting all ops after (but not including) 2
I20240426 09:55:31.431890 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:31.436275 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:31.492870 10056 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:31.360074 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:31.473132 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:31.436336 10090 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
W20240426 09:55:31.542332 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 49537.2us
I20240426 09:55:31.492599 9934 pending_rounds.cc:97] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9: Aborting uncommitted NO_OP operation due to leader change: 3.3
W20240426 09:55:31.399703 11723 consensus_meta.cc:287] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.509s user 0.009s sys 0.000s
I20240426 09:55:31.545265 9934 raft_consensus.cc:2866] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:31.569250 11723 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.283429 11513 raft_consensus.cc:3037] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 5
I20240426 09:55:31.484524 11721 raft_consensus.cc:3032] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:31.570996 11721 raft_consensus.cc:729] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:31.398046 11757 consensus_queue.cc:1046] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.167s
I20240426 09:55:31.571769 11721 consensus_queue.cc:260] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.572803 11721 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:31.354851 11739 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.576535 11739 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:31.566071 11872 log.cc:924] Time spent T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.380s user 0.001s sys 0.000s
I20240426 09:55:31.485028 11688 raft_consensus.cc:2726] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:31.374209 11812 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:31.568197 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 396839us
W20240426 09:55:31.485173 11871 log.cc:924] Time spent T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.325s user 0.000s sys 0.001s
I20240426 09:55:31.372720 11782 consensus_queue.cc:1046] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.196s
I20240426 09:55:31.482792 11804 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:31.436532 11846 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:31.595877 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:31.347388 11809 raft_consensus.cc:729] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:31.373817 11805 consensus_queue.cc:237] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:31.603940 11825 raft_consensus.cc:2726] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:31.635947 10007 raft_consensus.cc:2379] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 6: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:31.430681 11417 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:31.572744 11757 raft_consensus.cc:3032] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:31.527650 11697 raft_consensus.cc:2726] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:31.670150 11757 raft_consensus.cc:729] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:31.660823 11846 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:31.673992 11757 consensus_queue.cc:260] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.675153 11757 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:31.675604 11846 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:31.677273 11680 consensus_meta.cc:287] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.509s user 0.008s sys 0.000s
I20240426 09:55:31.677670 11680 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.399355 11829 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:31.640408 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:31.640035 9979 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:31.322075 11769 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:31.578393 11691 raft_consensus.cc:2726] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:31.482662 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:31.601308 11504 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:31.669582 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:31.681437 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:31.682003 9936 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.000s sys 0.017s
I20240426 09:55:31.686347 10082 raft_consensus.cc:1226] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:31.686846 9937 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:31.686868 10087 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
W20240426 09:55:31.687247 9936 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s user 0.000s sys 0.018s
I20240426 09:55:31.656062 11877 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:31.693519 9904 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:31.698062 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:31.681869 11644 raft_consensus.cc:3032] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:31.680051 11774 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:31.680191 11810 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:31.658733 11804 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.688724 10616 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:55:31.707779 10013 consensus_meta.cc:287] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.576s user 0.004s sys 0.004s
I20240426 09:55:31.685284 11723 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:31.719218 10013 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:31.680478 11417 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:31.707386 11644 raft_consensus.cc:729] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:31.735123 9938 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:31.735791 10009 raft_consensus.cc:1226] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 5. Current term is 6. Ops: []
I20240426 09:55:31.691484 11769 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.722568 11502 raft_consensus.cc:2741] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20240426 09:55:31.725788 11825 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:31.737041 11680 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:31.716140 11774 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:31.703004 11877 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:31.752521 11513 raft_consensus.cc:2726] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5.
I20240426 09:55:31.726926 11809 consensus_queue.cc:260] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.756595 10618 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:31.774919 11680 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:31.767366 11825 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:31.814458 11884 log.cc:924] Time spent T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.104s user 0.001s sys 0.000s
W20240426 09:55:31.814853 11889 log.cc:924] Time spent T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20240426 09:55:31.816327 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:31.816941 9940 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 6.
W20240426 09:55:31.807857 11886 log.cc:924] Time spent T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.146s user 0.000s sys 0.001s
I20240426 09:55:31.820070 11825 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:31.807693 11880 log.cc:924] Time spent T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.166s user 0.001s sys 0.000s
I20240426 09:55:31.824008 11877 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:31.818734 9939 raft_consensus.cc:2454] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:31.829459 9932 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:31.830427 11417 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:31.830667 9939 raft_consensus.cc:1226] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 2. Current term is 3. Ops: []
W20240426 09:55:31.832835 11881 log.cc:924] Time spent T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
W20240426 09:55:31.833173 11885 log.cc:924] Time spent T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.147s user 0.000s sys 0.000s
W20240426 09:55:31.833534 11879 log.cc:924] Time spent T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.149s user 0.001s sys 0.000s
W20240426 09:55:31.837663 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 144227us
I20240426 09:55:31.837733 11646 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:31.839100 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:31.840137 11782 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:31.840915 11462 consensus_meta.cc:287] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.636s user 0.005s sys 0.002s
I20240426 09:55:31.841436 11804 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:31.841810 10586 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:31.842381 11774 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:31.812222 11680 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:31.811748 11809 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:31.852209 10082 raft_consensus.cc:1261] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:31.849319 10090 raft_consensus.cc:1261] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 09:55:31.882941 10014 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 5.
W20240426 09:55:31.903425 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 61721.6us
I20240426 09:55:31.908066 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:31.902664 11875 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:31.794525 11644 consensus_queue.cc:260] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.907354 11462 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.923573 10085 raft_consensus.cc:2396] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 09:55:31.900121 11646 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:31.925134 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 381003us
I20240426 09:55:31.916992 11417 raft_consensus.cc:2781] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:55:31.927451 10585 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:31.900945 11504 consensus_queue.cc:1046] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20240426 09:55:31.928723 11646 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:31.965512 10056 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:31.904585 11782 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.970957 11473 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:31.971526 11473 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.973397 11473 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:31.975198 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:31.976076 10613 raft_consensus.cc:2379] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:31.932963 11417 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:31.979043 11417 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:31.963482 11628 raft_consensus.cc:2781] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:31.982854 11628 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:31.983211 11628 raft_consensus.cc:3037] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:31.988075 11680 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:31.990281 10056 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:31.992373 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:31.993162 10085 raft_consensus.cc:2379] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 3: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:31.927268 11893 consensus_queue.cc:1035] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.070s
I20240426 09:55:31.994792 11900 raft_consensus.cc:2781] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:31.995375 11900 raft_consensus.cc:686] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:31.996601 10616 raft_consensus.cc:2454] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:31.996280 11900 consensus_queue.cc:237] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.932585 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:31.998826 11774 raft_consensus.cc:2781] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:31.999353 11774 raft_consensus.cc:686] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
W20240426 09:55:31.999665 11876 log.cc:924] Time spent T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.352s user 0.000s sys 0.007s
I20240426 09:55:31.999984 11782 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:32.000134 11774 consensus_queue.cc:237] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.991021 11805 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:32.001257 11805 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:32.002140 11896 log.cc:924] Time spent T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20240426 09:55:32.003314 11805 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:31.990705 11769 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:32.004094 11898 log.cc:924] Time spent T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20240426 09:55:32.005854 11895 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:32.006126 11771 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:32.006326 11895 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.006649 11771 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.008286 11895 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:32.008425 11771 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:32.009255 11462 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:32.009781 11462 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:32.010191 11462 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:31.937920 11858 consensus_queue.cc:1035] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
I20240426 09:55:32.011384 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
W20240426 09:55:32.012002 11885 log.cc:924] Time spent T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
W20240426 09:55:32.015012 11582 consensus_meta.cc:287] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.785s user 0.005s sys 0.005s
I20240426 09:55:31.925228 11875 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.020155 11810 raft_consensus.cc:3032] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:32.022473 11473 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:32.023468 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:32.054128 9936 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:32.024152 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:32.055063 9932 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:32.024750 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:32.022204 10056 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:32.056039 9939 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
W20240426 09:55:32.001355 11882 log.cc:924] Time spent T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.517s user 0.000s sys 0.007s
I20240426 09:55:32.036453 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1344) took 1042 ms. Trace:
I20240426 09:55:32.045334 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:32.058399 10056 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:32.058492 9934 rpcz_store.cc:276] 0426 09:55:30.994454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:30.994559 (+ 105us) service_pool.cc:225] Handling call
0426 09:55:31.008483 (+ 13924us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:31.570808 (+562325us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:32.036200 (+465392us) spinlock_profiling.cc:243] Waited 344 ms on lock 0x7b5400261bd4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:32.036384 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":790117248}
I20240426 09:55:32.059257 10056 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:32.058566 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:32.064191 10081 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:32.060005 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:32.065047 10015 raft_consensus.cc:3037] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:32.067039 9904 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:32.071744 11646 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.072271 10618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45564 (request call id 239) took 1347 ms. Trace:
I20240426 09:55:32.085680 10618 rpcz_store.cc:276] 0426 09:55:30.725209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:30.726741 (+ 1532us) service_pool.cc:225] Handling call
0426 09:55:32.072087 (+1345346us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b54005055d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:32.072198 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1755995520}
I20240426 09:55:32.086850 11646 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:32.072955 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:32.074028 11769 raft_consensus.cc:2781] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:32.088675 11679 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:32.089097 11769 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:32.089566 11769 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:32.090029 11825 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:32.090536 11825 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:32.092051 10055 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:32.092401 11825 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:32.092866 11825 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:32.093282 11825 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.093653 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:32.062372 11828 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:32.094594 11828 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:32.094987 11828 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:32.095786 11454 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:32.074440 11877 consensus_queue.cc:1035] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:32.080580 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:32.082187 11462 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:32.068938 11805 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:32.084122 11473 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:32.087152 11757 raft_consensus.cc:2781] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:32.107722 11757 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:32.108093 11757 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:32.108575 10016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1064) took 1141 ms. Trace:
I20240426 09:55:32.016144 11417 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:32.089219 11679 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.072736 11680 raft_consensus.cc:2781] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:32.111196 11680 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:32.111591 11680 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:32.096835 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:32.050966 11782 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:32.099129 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:32.099906 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:32.053601 11810 raft_consensus.cc:729] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:32.102183 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:32.102424 10055 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:32.104452 9935 raft_consensus.cc:2379] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:32.032063 11628 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:32.063364 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:32.089705 11644 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:32.106227 11805 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.106917 11895 raft_consensus.cc:2726] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:32.107308 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1067) took 1079 ms. Trace:
I20240426 09:55:32.108915 10016 rpcz_store.cc:276] 0426 09:55:30.967699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:30.967830 (+ 131us) service_pool.cc:225] Handling call
0426 09:55:30.976639 (+ 8809us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:30.976808 (+ 169us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 09:55:31.087961 (+111153us) log.cc:842] Serialized 14 byte log entry
0426 09:55:31.431568 (+343607us) log.cc:842] Serialized 14 byte log entry
0426 09:55:31.830071 (+398503us) spinlock_profiling.cc:243] Waited 3.97 ms on lock 0x7b180027cc40. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fce97c6 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fd1be78 00007f9c9fd1f351
0426 09:55:31.830560 (+ 489us) spinlock_profiling.cc:243] Waited 30.6 ms on lock 0x7b180027cc40. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fd1be78 00007f9c9fd1f351 00007f9c9fd1f309 00007f9c9fd1f299 00007f9c9fd1f261 00007f9c9fd1e55d 00007f9ca8f8fd34 00007f9ca8f8fb69 00007f9c9fcfe5f6
0426 09:55:32.108159 (+277599us) spinlock_profiling.cc:243] Waited 30.6 ms on lock 0x7b180027cc40. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fc94faa 00007f9ca9383f25 00007f9c9fd19094 00007f9c9fd18985 00007f9ca2c3e560 00007f9ca2c50706 00007f9ca2c61267 00007f9ca2c611e8 00007f9ca2c61168
0426 09:55:32.108287 (+ 128us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:32.108329 (+ 42us) raft_consensus.cc:1631] Marking committed up to 4
0426 09:55:32.108366 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:32.108423 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:32.108527 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4263,"prepare.queue_time_us":60181,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":4656,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":12923,"spinlock_wait_cycles":150003072,"thread_start_us":447131,"threads_started":2,"wal-append.queue_time_us":458799}
I20240426 09:55:32.112257 10087 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
I20240426 09:55:32.079456 11875 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:32.094208 10085 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 5.
I20240426 09:55:32.096276 11454 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.125623 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:32.138099 10082 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 7
I20240426 09:55:32.133782 9932 raft_consensus.cc:2454] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:32.135481 10009 rpcz_store.cc:276] 0426 09:55:31.027516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:31.027673 (+ 157us) service_pool.cc:225] Handling call
0426 09:55:31.735640 (+707967us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:31.736192 (+ 552us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:32.107141 (+370949us) spinlock_profiling.cc:243] Waited 690 ms on lock 0x7b540019a5d0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c45b99 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:32.107259 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1587706112}
I20240426 09:55:32.134052 9980 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:32.145612 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:32.146371 9935 raft_consensus.cc:2379] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
W20240426 09:55:32.147235 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 507248us
I20240426 09:55:32.149154 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:32.149492 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:32.150226 9939 raft_consensus.cc:2379] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 6: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:32.159490 9936 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:32.160663 10056 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:32.164464 10015 raft_consensus.cc:2454] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:32.123369 11462 raft_consensus.cc:2726] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:55:32.114759 11771 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:32.180217 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:32.180856 9939 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:32.182022 10056 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:32.178203 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:32.133544 11679 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:32.183719 9936 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:32.123207 11417 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:32.185865 10082 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 7.
I20240426 09:55:32.187674 10583 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:32.143169 10613 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:32.174746 11533 raft_consensus.cc:2726] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:32.190831 11771 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.207706 11740 raft_consensus.cc:2781] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:32.208209 11740 raft_consensus.cc:686] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 7 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:32.177320 11875 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:32.192874 9982 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:32.190865 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:32.212095 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:32.193174 11502 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:32.193620 11693 raft_consensus.cc:2781] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election won for term 4
W20240426 09:55:32.193569 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 59588.8us
I20240426 09:55:32.218607 9980 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:32.219050 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:32.176847 11545 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20240426 09:55:32.135754 11782 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.195317 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:32.195983 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.200558 10611 raft_consensus.cc:2354] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 5. Current term is 6.
I20240426 09:55:32.200806 11829 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:32.201575 11688 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:32.201211 11810 consensus_queue.cc:260] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.202597 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:32.219556 9939 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:32.220312 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:32.194043 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
W20240426 09:55:32.190155 11814 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.620s user 0.004s sys 0.014s
I20240426 09:55:32.211436 9935 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:32.227043 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:32.227754 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.228973 10016 raft_consensus.cc:2379] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:32.226646 11408 raft_consensus.cc:2781] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:32.209060 11875 raft_consensus.cc:686] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:32.188676 11825 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:32.233909 9941 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:32.237111 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:32.246265 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.233258 11696 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:32.277518 11696 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:32.278024 9935 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:32.279345 9935 raft_consensus.cc:1261] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:32.280357 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:32.281061 10086 raft_consensus.cc:2379] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 4: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:32.281579 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.281684 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:32.256759 10088 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
W20240426 09:55:32.256099 11814 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.691s user 0.004s sys 0.014s
I20240426 09:55:32.248845 11688 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.247041 11582 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:32.231946 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:32.239507 11729 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.997s user 0.013s sys 0.035s
I20240426 09:55:32.231073 11739 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:32.262472 10056 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:32.270938 9934 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:32.271889 11875 consensus_queue.cc:237] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.277907 11696 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:32.272480 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:32.273064 11828 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.209013 11740 consensus_queue.cc:237] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:32.221180 11502 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.282546 11454 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:32.274816 11810 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:32.282711 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:32.306907 11408 raft_consensus.cc:686] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:32.230736 11693 raft_consensus.cc:686] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:32.305747 11454 raft_consensus.cc:2781] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:32.283102 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:32.288293 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:32.309391 10082 raft_consensus.cc:2379] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 6: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:32.293310 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.310796 10011 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:32.295284 9939 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 7.
I20240426 09:55:32.295850 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:32.295998 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.296447 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.297767 9937 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:32.298470 10083 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:32.300662 10056 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:32.252713 9936 raft_consensus.cc:2454] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:32.304886 10008 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240426 09:55:32.316483 9940 raft_consensus.cc:1226] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240426 09:55:32.292292 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.316606 10008 pending_rounds.cc:76] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting all ops after (but not including) 2
I20240426 09:55:32.318028 10008 pending_rounds.cc:97] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240426 09:55:32.318284 10008 raft_consensus.cc:2866] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:32.295143 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.327132 10007 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:32.327451 10083 raft_consensus.cc:1261] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 09:55:32.330627 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:32.331378 10083 raft_consensus.cc:3037] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 6
I20240426 09:55:32.395915 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:32.396716 9941 raft_consensus.cc:2379] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:32.397117 10085 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:32.397882 10087 raft_consensus.cc:2454] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
W20240426 09:55:32.398124 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 129262us
I20240426 09:55:32.401060 10083 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 6.
W20240426 09:55:32.395217 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 69856.2us
W20240426 09:55:32.295536 11729 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.064s user 0.014s sys 0.035s
I20240426 09:55:32.305621 11502 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:32.290205 11902 log.cc:924] Time spent T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.117s user 0.001s sys 0.000s
I20240426 09:55:32.411131 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:32.420405 10007 raft_consensus.cc:3037] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:32.412772 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.421161 10013 raft_consensus.cc:2379] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:32.413780 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.296157 11782 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:32.302003 11828 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:32.282794 11771 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:32.426858 9904 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:32.417680 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.418354 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:32.299786 11473 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:32.422001 10011 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:32.407544 10055 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:32.295686 11739 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:32.328017 11454 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:32.285554 11688 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:32.299873 11628 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:32.395082 11733 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:32.433822 10054 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:32.281864 11680 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.434026 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:32.415221 11900 raft_consensus.cc:2781] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:32.434859 11807 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:32.471396 11900 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:32.471582 11807 raft_consensus.cc:686] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:32.472426 11807 consensus_queue.cc:237] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.434053 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:32.307747 11408 consensus_queue.cc:237] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:32.439148 11693 consensus_queue.cc:237] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:32.478463 10084 raft_consensus.cc:1226] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:32.440447 11805 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:32.440946 11828 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:32.441298 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:32.283643 11769 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:32.238394 11829 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:32.291610 11757 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:32.288797 11582 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.450842 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:55:32.453905 11883 log.cc:924] Time spent T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.728s user 0.017s sys 0.024s
I20240426 09:55:32.305130 11696 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:32.457374 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 49906.2us
I20240426 09:55:32.460124 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:32.467144 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:32.470160 11680 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:32.470638 11697 raft_consensus.cc:2726] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:32.416505 11774 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:32.482699 11733 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.451951 11454 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:32.452275 11739 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:32.581475 10055 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:32.489331 11900 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:32.598318 9905 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:32.470583 11688 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 09:55:32.587652 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 84738.2us
I20240426 09:55:32.594241 10012 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:32.597404 10083 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:32.600382 10055 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:32.619107 10055 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
W20240426 09:55:32.617350 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 432305us
W20240426 09:55:32.642333 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 382516us
I20240426 09:55:32.616580 10616 raft_consensus.cc:1226] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:32.609606 11697 raft_consensus.cc:2726] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:32.654616 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.655364 11733 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:32.634745 11858 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:32.655550 11778 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:32.671615 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:32.671485 11778 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.672245 9940 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:32.656140 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.655282 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.660575 11829 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:32.661092 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:32.625706 11502 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:32.663604 11582 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:32.597693 11774 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:32.668759 10089 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 1007) took 1520 ms. Trace:
I20240426 09:55:32.676688 10015 raft_consensus.cc:2379] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:32.677258 10013 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:32.632830 11771 raft_consensus.cc:2781] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:32.639889 11895 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:32.673404 11778 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:32.677870 10007 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 6.
W20240426 09:55:32.678061 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 255015us
I20240426 09:55:32.674679 11696 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:32.694784 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.679433 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.696411 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.696923 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.688884 10089 rpcz_store.cc:276] 0426 09:55:31.148382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:31.148514 (+ 132us) service_pool.cc:225] Handling call
0426 09:55:31.155665 (+ 7151us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 09:55:31.170807 (+ 15142us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:31.170836 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 09:55:31.177455 (+ 6619us) log.cc:842] Serialized 52 byte log entry
0426 09:55:31.374095 (+196640us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:31.466961 (+ 92866us) log.cc:842] Serialized 14 byte log entry
0426 09:55:31.483165 (+ 16204us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:31.483233 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:31.827971 (+344738us) spinlock_profiling.cc:243] Waited 36.3 ms on lock 0x7b18003150a0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fd1bfa4 00007f9c9fd1f351 00007f9c9fd1f309 00007f9c9fd1f299 00007f9c9fd1f261 00007f9c9fd1e55d 00007f9ca8f8fd34 00007f9ca8f8fb69 00007f9c9fcfe5f6
0426 09:55:32.453381 (+625410us) spinlock_profiling.cc:243] Waited 6.57 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:55:32.668574 (+215193us) raft_consensus.cc:1671] finished
0426 09:55:32.668593 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:32.668711 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20191,"prepare.queue_time_us":92834,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":6958,"raft.queue_time_us":26871,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":51379,"spinlock_wait_cycles":98583296,"thread_start_us":134520,"threads_started":1,"wal-append.queue_time_us":495285}
I20240426 09:55:32.696107 10056 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:32.680450 10016 raft_consensus.cc:1261] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 09:55:32.698390 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:32.699540 10015 raft_consensus.cc:2454] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:32.700076 10056 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:32.701267 10007 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:32.701378 10056 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:32.693104 10585 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:32.707767 10056 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
W20240426 09:55:32.709232 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 92007.5us
W20240426 09:55:32.833727 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 126014us
I20240426 09:55:32.772578 11875 raft_consensus.cc:2781] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:32.773109 11830 raft_consensus.cc:2726] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:32.802542 10013 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:32.803017 10007 raft_consensus.cc:1226] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: []
W20240426 09:55:32.803369 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 182120us
I20240426 09:55:32.708518 11696 raft_consensus.cc:2726] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:32.806881 10015 raft_consensus.cc:1261] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240426 09:55:32.803857 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 116306us
I20240426 09:55:32.691663 11858 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:32.806288 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:32.845929 10011 raft_consensus.cc:2379] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:32.811697 10016 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:32.690130 11771 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:32.690318 11895 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:32.678145 11825 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:32.839485 10055 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
W20240426 09:55:32.826570 11740 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s user 0.018s sys 0.009s
I20240426 09:55:32.803784 11861 raft_consensus.cc:2781] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:32.846704 10016 raft_consensus.cc:595] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:32.803705 11644 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:32.691408 11502 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:32.853853 10055 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
W20240426 09:55:32.864279 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 201970us
I20240426 09:55:32.839267 11408 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:32.865301 9906 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:32.867959 9982 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:32.868809 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:32.871726 10086 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:32.850394 11771 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:32.853739 11895 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:55:32.928525 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 56465.7us
W20240426 09:55:32.926470 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 57064.7us
I20240426 09:55:32.850471 11757 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:32.864040 11829 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:32.863198 11769 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:32.853068 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:32.937865 9906 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:32.984692 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 321667us
I20240426 09:55:32.870461 11861 raft_consensus.cc:686] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 6 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:32.926573 11891 raft_consensus.cc:2726] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:32.841065 11805 consensus_queue.cc:1046] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.095s
I20240426 09:55:32.939469 11830 raft_consensus.cc:2781] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:32.852412 11858 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:32.994019 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 56284.3us
I20240426 09:55:32.870572 11644 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:32.870199 11825 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:33.000295 9979 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:32.945169 11829 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:55:33.120054 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 119845us
I20240426 09:55:32.927631 11408 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:33.122663 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 62908.9us
W20240426 09:55:33.047633 11915 log.cc:924] Time spent T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.356s user 0.001s sys 0.000s
W20240426 09:55:33.044461 11918 log.cc:924] Time spent T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.340s user 0.000s sys 0.001s
W20240426 09:55:33.044838 11930 log.cc:924] Time spent T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.207s user 0.000s sys 0.000s
W20240426 09:55:33.047832 11919 log.cc:924] Time spent T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.338s user 0.001s sys 0.000s
W20240426 09:55:33.048023 11931 log.cc:924] Time spent T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.092s user 0.000s sys 0.001s
I20240426 09:55:33.041639 11830 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:55:33.127697 11454 consensus_meta.cc:287] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.511s user 0.008s sys 0.000s
W20240426 09:55:33.125233 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 254360us
I20240426 09:55:33.127380 9979 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:33.126475 9906 leader_election.cc:398] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4.
I20240426 09:55:32.997974 11890 raft_consensus.cc:2726] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:33.141155 11934 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:33.141840 11857 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:33.045805 11936 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:32.986224 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
W20240426 09:55:33.144325 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 276414us
W20240426 09:55:33.123128 11925 log.cc:924] Time spent T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.078s user 0.000s sys 0.002s
I20240426 09:55:33.047420 11927 raft_consensus.cc:2726] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:55:33.051235 11774 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:33.060396 11875 raft_consensus.cc:686] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:33.188175 11502 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:33.188736 11875 consensus_queue.cc:237] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:33.189338 11916 log.cc:924] Time spent T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.427s user 0.000s sys 0.001s
I20240426 09:55:33.134405 11829 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:32.998993 11693 consensus_queue.cc:1046] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.153s
W20240426 09:55:33.166441 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 221747us
I20240426 09:55:33.127446 11417 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:33.169031 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:33.040318 11922 consensus_queue.cc:1035] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.172s
I20240426 09:55:33.155812 11830 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:33.155277 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:33.166971 11454 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.191392 10081 raft_consensus.cc:1261] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:33.192432 10090 raft_consensus.cc:1226] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20240426 09:55:33.172243 11934 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:33.186946 11812 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:33.170861 11936 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:33.200584 10618 raft_consensus.cc:2379] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:33.189218 11502 raft_consensus.cc:2726] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:33.166834 11805 consensus_queue.cc:1035] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
W20240426 09:55:33.215726 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 88634.1us
I20240426 09:55:33.216468 9979 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:33.217025 10013 raft_consensus.cc:1261] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:33.179565 11857 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:33.228138 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:33.228730 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:33.229334 10016 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:33.229694 11947 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:33.230149 11947 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.230393 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:33.231011 10617 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:33.231173 11825 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:33.231535 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:33.232065 11947 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:33.204785 11417 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:33.233289 11454 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:33.234376 11928 log.cc:924] Time spent T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.240s user 0.000s sys 0.001s
I20240426 09:55:33.232216 10010 raft_consensus.cc:2379] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:33.229292 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:33.237357 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:33.240892 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:33.242399 9939 raft_consensus.cc:2454] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:33.237998 9938 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:33.241166 10081 raft_consensus.cc:1261] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 09:55:33.239784 10055 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:33.208878 11936 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:33.206780 11948 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:33.246279 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:33.247854 9940 raft_consensus.cc:1261] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20240426 09:55:33.225518 11914 log.cc:924] Time spent T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.384s user 0.001s sys 0.000s
I20240426 09:55:33.171864 11861 consensus_queue.cc:237] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:33.250957 9979 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:33.253150 9982 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:33.254120 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:33.254501 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:33.254540 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:33.234669 11739 consensus_meta.cc:287] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.609s user 0.004s sys 0.004s
W20240426 09:55:33.240005 11923 log.cc:924] Time spent T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.078s user 0.000s sys 0.001s
W20240426 09:55:33.239986 11929 log.cc:924] Time spent T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.116s user 0.005s sys 0.003s
I20240426 09:55:33.223372 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:33.263207 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:33.210218 11812 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:33.282224 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 69777.5us
I20240426 09:55:33.265754 10086 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:33.284152 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1373) took 1026 ms. Trace:
I20240426 09:55:33.284550 10015 raft_consensus.cc:1261] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:33.248435 11417 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:33.286726 9935 rpcz_store.cc:276] 0426 09:55:32.257962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:32.286635 (+ 28673us) service_pool.cc:225] Handling call
0426 09:55:32.286920 (+ 285us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:32.418480 (+131560us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:32.418507 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:32.418903 (+ 396us) log.cc:842] Serialized 26 byte log entry
0426 09:55:32.679619 (+260716us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:32.680142 (+ 523us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:32.680205 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:33.283962 (+603757us) raft_consensus.cc:1671] finished
0426 09:55:33.283979 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:33.284097 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":143185,"raft.queue_time_us":128098,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":24449,"thread_start_us":242010,"threads_started":1,"wal-append.queue_time_us":443003}
I20240426 09:55:33.290673 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:33.312989 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 73291.8us
I20240426 09:55:33.314440 10055 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:33.256675 11502 raft_consensus.cc:3032] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:33.289844 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:33.315304 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:33.317777 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 77578.9us
W20240426 09:55:33.289615 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 455179us
I20240426 09:55:33.312386 10086 raft_consensus.cc:2454] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:33.314162 9904 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:33.241365 11938 consensus_queue.cc:1035] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20240426 09:55:33.328915 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:33.330824 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:55:33.335886 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 80765.2us
I20240426 09:55:33.337242 10615 raft_consensus.cc:1226] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20240426 09:55:33.337186 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:33.281785 11644 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:33.333432 9934 raft_consensus.cc:1261] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:33.317458 10616 raft_consensus.cc:2379] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:33.311376 11408 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:33.315690 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:33.342572 11949 raft_consensus.cc:2726] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:33.317795 11502 raft_consensus.cc:729] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:33.343369 9941 raft_consensus.cc:1261] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:33.343848 10611 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45570 (request call id 183) took 1442 ms. Trace:
I20240426 09:55:33.344269 11937 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:33.343847 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:33.347201 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1410) took 1054 ms. Trace:
I20240426 09:55:33.323217 11825 raft_consensus.cc:2781] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:33.344085 11955 raft_consensus.cc:2726] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:33.281667 11739 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.332405 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:33.260304 11948 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.326587 11858 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:33.339046 11805 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:33.360837 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 107791us
I20240426 09:55:33.317380 11769 raft_consensus.cc:2726] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:33.376662 9932 raft_consensus.cc:2354] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 4. Current term is 5.
W20240426 09:55:33.384399 11935 log.cc:924] Time spent T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.204s user 0.000s sys 0.000s
I20240426 09:55:33.376513 10008 rpcz_store.cc:276] 0426 09:55:32.292547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:32.295230 (+ 2683us) service_pool.cc:225] Handling call
0426 09:55:32.295472 (+ 242us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:32.330906 (+ 35434us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:33.346997 (+1016091us) spinlock_profiling.cc:243] Waited 9.09 ms on lock 0x7b5400056ed4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:33.347151 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20902144}
W20240426 09:55:33.394253 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 64267.1us
I20240426 09:55:33.397397 10586 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:33.394943 9940 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:33.398453 9982 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:33.335625 11417 consensus_queue.cc:1035] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.035s
W20240426 09:55:33.400116 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 113068us
I20240426 09:55:33.401252 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:33.406883 10009 raft_consensus.cc:2379] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 6: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:33.355772 10611 rpcz_store.cc:276] 0426 09:55:31.923773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:31.926061 (+ 2288us) service_pool.cc:225] Handling call
0426 09:55:33.343659 (+1417598us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b54006004d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:33.343792 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":366749440}
I20240426 09:55:33.426302 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:33.436483 11857 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:33.436664 10613 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:33.436774 11943 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:33.437055 11857 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:33.437362 11943 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.438617 9980 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:33.441092 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:33.386456 11825 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:33.441792 10008 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:55:33.455164 10585 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:33.458323 11944 raft_consensus.cc:2781] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:33.401017 11454 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:33.438633 10615 raft_consensus.cc:1261] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 09:55:33.461009 11964 raft_consensus.cc:2781] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:33.461458 11964 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:33.461783 11964 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:55:33.464282 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 140729us
I20240426 09:55:33.458875 11955 raft_consensus.cc:2781] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:33.401531 10618 raft_consensus.cc:2379] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
W20240426 09:55:33.467096 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 68719.6us
I20240426 09:55:33.482137 11921 raft_consensus.cc:2741] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20240426 09:55:33.456930 11825 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
W20240426 09:55:33.468267 11883 log.cc:924] Time spent T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20240426 09:55:33.465487 11944 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:33.492178 9982 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
W20240426 09:55:33.493981 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 143093us
I20240426 09:55:33.498512 11958 consensus_queue.cc:1035] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20240426 09:55:33.470584 11934 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:33.473985 11955 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:33.387135 11937 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.500161 11967 raft_consensus.cc:2781] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:33.516703 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:33.518177 11963 consensus_queue.cc:1035] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20240426 09:55:33.518999 11961 consensus_queue.cc:1035] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.011s
I20240426 09:55:33.520299 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:33.501358 11944 raft_consensus.cc:3037] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:33.466302 11454 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.537572 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:33.525523 11976 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 09:55:33.459237 11858 consensus_queue.cc:1046] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20240426 09:55:33.439121 11502 consensus_queue.cc:260] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.535717 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:33.402017 11805 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:33.544382 11953 log.cc:924] Time spent T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20240426 09:55:33.512419 11943 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:33.522265 11958 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:33.529135 11545 raft_consensus.cc:2726] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:33.520038 11955 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:33.537508 11967 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:33.538465 10616 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 09:55:33.537765 11812 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:33.568552 9940 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
W20240426 09:55:33.574502 11771 consensus_meta.cc:287] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.619s user 0.005s sys 0.002s
I20240426 09:55:33.570511 11502 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:33.572316 10617 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:33.567911 11967 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
W20240426 09:55:33.605116 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 351094us
W20240426 09:55:33.606889 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 90767.6us
I20240426 09:55:33.608691 10583 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:33.584204 11739 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:33.574594 11936 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.610131 9979 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:33.615224 9979 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:33.605549 11948 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:33.569697 11958 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.594643 11771 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.611999 11940 raft_consensus.cc:2726] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:33.610602 11741 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:33.637270 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 239374us
I20240426 09:55:33.637965 9904 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:33.618371 9979 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:33.640533 11977 raft_consensus.cc:2781] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election won for term 5
W20240426 09:55:33.641148 11900 consensus_meta.cc:287] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 1.014s user 0.007s sys 0.003s
I20240426 09:55:33.619521 11778 raft_consensus.cc:2781] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:33.590512 11825 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:33.640877 10015 raft_consensus.cc:1226] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240426 09:55:33.643172 9904 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:33.616887 10054 leader_election.cc:398] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Vote denied by peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313) with higher term. Message: Invalid argument: T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 5. Current term is 6.
I20240426 09:55:33.647543 10054 leader_election.cc:401] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20240426 09:55:33.639482 11741 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.648782 10089 raft_consensus.cc:1226] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:33.649705 10054 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:33.651850 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1145) took 2282 ms. Trace:
I20240426 09:55:33.619846 11739 consensus_queue.cc:1035] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20240426 09:55:33.652240 9933 rpcz_store.cc:276] 0426 09:55:31.363331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:31.539297 (+175966us) service_pool.cc:225] Handling call
0426 09:55:32.010606 (+471309us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 09:55:32.023825 (+ 13219us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 09:55:32.024102 (+ 277us) log.cc:842] Serialized 14 byte log entry
0426 09:55:32.244256 (+220154us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 09:55:32.269463 (+ 25207us) log.cc:842] Serialized 52 byte log entry
0426 09:55:32.269576 (+ 113us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:55:32.474201 (+204625us) spinlock_profiling.cc:243] Waited 62.8 ms on lock 0x7b1800437080. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fc94faa 00007f9ca9383f25 00007f9c9fd19094 00007f9c9fd18985 00007f9ca661cc07 00007f9ca2c50997 00007f9ca2c6009f 00007f9ca2c60018 00007f9ca2c5ff98
0426 09:55:32.474390 (+ 189us) log.cc:842] Serialized 14 byte log entry
0426 09:55:32.474564 (+ 174us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:32.474627 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:32.474740 (+ 113us) raft_consensus.cc:1671] finished
0426 09:55:32.474754 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:33.645582 (+1170828us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b54000bf750. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c45b99 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:33.645690 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":144503,"prepare.queue_time_us":205775,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":73156,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":13935,"spinlock_wait_cycles":965167232,"thread_start_us":165916,"threads_started":1,"wal-append.queue_time_us":236622,"wal-append.run_cpu_time_us":2384,"wal-append.run_wall_time_us":1200411}
I20240426 09:55:33.615578 11956 raft_consensus.cc:2726] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:33.658861 9938 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:33.659559 10086 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:55:33.653723 11454 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:33.648690 11963 raft_consensus.cc:2781] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:33.664937 11741 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:33.665158 11696 consensus_queue.cc:1046] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:33.666040 11696 raft_consensus.cc:3032] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:33.666373 11696 raft_consensus.cc:729] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:33.667192 11696 consensus_queue.cc:260] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.668313 11696 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:33.663259 11948 consensus_queue.cc:1046] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:33.669994 11948 raft_consensus.cc:3032] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:33.670310 11948 raft_consensus.cc:729] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:33.671072 11948 consensus_queue.cc:260] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.671573 11936 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:33.672286 11948 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:55:33.674499 11895 consensus_meta.cc:287] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.720s user 0.007s sys 0.000s
I20240426 09:55:33.675228 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:33.684283 10015 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:33.676388 11644 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:33.651170 11778 raft_consensus.cc:686] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
W20240426 09:55:33.689069 11962 log.cc:924] Time spent T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.133s user 0.000s sys 0.001s
W20240426 09:55:33.689433 9937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 1151) took 2047 ms (client timeout 2240 ms). Trace:
I20240426 09:55:33.641121 11977 raft_consensus.cc:686] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:33.687131 10055 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
W20240426 09:55:33.689806 9937 rpcz_store.cc:269] 0426 09:55:31.639953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:31.687823 (+ 47870us) service_pool.cc:225] Handling call
0426 09:55:33.686830 (+1999007us) spinlock_profiling.cc:243] Waited 590 ms on lock 0x7b54001aed54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:33.686927 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1356879232}
I20240426 09:55:33.651736 11937 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:33.692698 11983 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
W20240426 09:55:33.693737 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 50625.2us
I20240426 09:55:33.656546 11809 raft_consensus.cc:465] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Not starting pre-election -- already a leader
I20240426 09:55:33.658623 11976 raft_consensus.cc:2726] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313) with higher term. Message: Invalid argument: T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 5. Current term is 6.
I20240426 09:55:33.664862 11963 raft_consensus.cc:686] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:33.646245 11900 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.702533 9906 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:33.694245 11875 raft_consensus.cc:2781] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:33.693248 11454 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:33.704744 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:33.704806 11454 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:33.706022 11646 raft_consensus.cc:2726] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:33.706804 11454 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:33.701328 11979 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:33.707671 11979 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:33.708447 11900 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:33.696276 11893 raft_consensus.cc:2781] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:33.709478 10056 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:33.715437 10085 raft_consensus.cc:1226] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6. Ops: []
I20240426 09:55:33.711268 9906 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:33.715823 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:33.711313 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:33.712553 10082 raft_consensus.cc:2379] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 5: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:33.712160 10088 raft_consensus.cc:1261] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:33.709749 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:33.716624 10086 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:33.717320 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:33.718799 11979 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:33.716956 10083 raft_consensus.cc:2454] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:33.720669 9906 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:33.721594 9937 raft_consensus.cc:3032] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:33.722563 9937 raft_consensus.cc:729] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:33.721599 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:33.723426 9982 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:33.722095 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:33.724481 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:33.724715 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:33.725768 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:33.725646 11769 consensus_queue.cc:1035] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:33.726238 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:33.726879 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:33.726717 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:33.728045 9934 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:33.722157 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:33.728403 9940 raft_consensus.cc:1226] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6. Ops: []
I20240426 09:55:33.719316 11454 raft_consensus.cc:2726] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:33.724483 9941 raft_consensus.cc:2379] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 6: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:33.729357 10056 leader_election.cc:398] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 4. Current term is 5.
I20240426 09:55:33.731923 10056 leader_election.cc:401] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20240426 09:55:33.732103 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:33.723764 9937 consensus_queue.cc:260] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:33.690860 11895 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.733085 9937 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:33.700737 11805 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:33.730518 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:33.735056 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:33.731164 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:33.735499 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:33.735754 9934 raft_consensus.cc:2454] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 2.
I20240426 09:55:33.736388 9906 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:33.735615 9940 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:33.737397 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:33.736524 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:33.738232 9906 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:33.739279 9982 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:33.727389 11979 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:33.714412 11893 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:33.741663 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:33.713769 11875 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:33.699805 11733 consensus_queue.cc:1035] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20240426 09:55:33.761381 10056 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:33.745445 9941 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:33.740300 11979 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.743577 11921 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:33.715127 11814 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:33.742050 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:33.751009 11972 raft_consensus.cc:2781] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:33.689754 11778 consensus_queue.cc:237] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.751844 11771 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:33.752707 9937 raft_consensus.cc:1261] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20240426 09:55:33.753522 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:55:33.758073 11829 consensus_meta.cc:287] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.560s user 0.003s sys 0.006s
I20240426 09:55:33.762576 10056 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:33.744253 11825 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:33.768142 11877 raft_consensus.cc:2781] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:33.736194 11473 raft_consensus.cc:2726] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:33.772982 9937 pending_rounds.cc:76] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9: Aborting all ops after (but not including) 5
I20240426 09:55:33.840268 9937 pending_rounds.cc:97] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20240426 09:55:33.748448 11875 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:33.745000 11741 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:33.704324 11830 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.740610 11868 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:33.740964 11646 raft_consensus.cc:2726] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:33.769191 11979 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:33.778045 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:33.748067 11893 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:33.840571 9937 raft_consensus.cc:2866] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 09:55:33.905864 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 167691us
I20240426 09:55:33.727013 11943 consensus_queue.cc:1035] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20240426 09:55:33.747876 11937 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:33.911005 11945 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:33.910912 11937 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.851442 11895 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:33.851991 11983 raft_consensus.cc:2726] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:33.918457 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1091) took 1265 ms. Trace:
I20240426 09:55:33.743494 11963 consensus_queue.cc:237] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.852481 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:33.769835 11454 raft_consensus.cc:2726] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 4. Current term is 5.
I20240426 09:55:33.918826 10007 rpcz_store.cc:276] 0426 09:55:32.652678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:32.701885 (+ 49207us) service_pool.cc:225] Handling call
0426 09:55:32.702144 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:32.841156 (+139012us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:33.918255 (+1077099us) spinlock_profiling.cc:243] Waited 25 ms on lock 0x7b540027e7d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:33.918405 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57525120}
I20240426 09:55:33.744711 11858 consensus_queue.cc:1035] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.098s
I20240426 09:55:33.773763 11921 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:33.871335 9940 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:33.776909 11693 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:33.777987 11972 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:33.839356 11968 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election won for term 5
W20240426 09:55:33.903414 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 163906us
W20240426 09:55:33.914363 11807 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20240426 09:55:33.912724 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:33.853246 11739 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:33.844794 11809 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:33.915278 11646 consensus_queue.cc:1035] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20240426 09:55:33.974045 11809 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.925925 11921 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:33.845996 11771 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:33.983103 10083 raft_consensus.cc:1261] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 09:55:33.855592 11857 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:33.764207 11977 consensus_queue.cc:237] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:33.917510 11937 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:33.853058 11230 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:33.776320 11814 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.921730 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:33.906812 11741 raft_consensus.cc:686] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:33.779248 11829 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.937877 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:34.005868 9937 raft_consensus.cc:2379] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:33.938235 11968 raft_consensus.cc:686] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:33.911545 11945 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:33.912051 11733 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:34.009202 11733 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.945596 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:33.945659 10014 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:33.908337 11688 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:33.982646 11739 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.988236 11857 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:33.993600 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:33.994591 11230 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.982367 11976 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:33.996088 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 57965.8us
I20240426 09:55:33.998219 9979 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:34.008196 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:33.927539 11972 raft_consensus.cc:3037] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:33.937814 11805 consensus_queue.cc:1035] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20240426 09:55:34.031697 9979 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:34.033183 10081 raft_consensus.cc:1226] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6. Ops: []
I20240426 09:55:34.035316 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:34.038121 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:34.044494 10087 raft_consensus.cc:2396] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240426 09:55:34.046149 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:34.046424 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:34.005115 11771 raft_consensus.cc:686] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:34.050561 11957 raft_consensus.cc:2726] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:55:33.954119 11693 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:33.903095 11868 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:34.062880 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:34.011102 11922 consensus_queue.cc:1046] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20240426 09:55:34.067818 10014 raft_consensus.cc:2379] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:34.080310 11921 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:34.082306 11921 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:33.946297 11964 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:34.032656 11688 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:34.069622 11944 consensus_meta.cc:287] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.525s user 0.005s sys 0.005s
I20240426 09:55:34.043628 11976 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:34.074988 11957 raft_consensus.cc:2726] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:34.103377 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:34.074585 11693 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:34.104753 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:34.106024 10083 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
W20240426 09:55:34.107122 11984 log.cc:924] Time spent T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.390s user 0.003s sys 0.000s
I20240426 09:55:34.107398 10617 raft_consensus.cc:2354] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4.
I20240426 09:55:34.106460 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:34.109774 9904 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:34.110317 10014 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:34.090754 11814 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:34.115787 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:34.116652 10008 raft_consensus.cc:2396] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 09:55:34.118371 11230 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:34.119957 11688 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:34.122644 10007 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:34.092669 11809 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:34.123979 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:34.034387 11955 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:34.104321 11830 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:34.133774 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:34.139662 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:34.140394 10008 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:34.141369 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:55:34.126385 11973 log.cc:924] Time spent T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.260s user 0.000s sys 0.001s
I20240426 09:55:34.146085 10011 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:34.153291 10013 raft_consensus.cc:2354] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4.
I20240426 09:55:34.127106 11858 raft_consensus.cc:2781] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:34.162027 11739 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:34.154589 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:34.097743 11944 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:34.092273 11741 consensus_queue.cc:237] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:34.114665 11771 consensus_queue.cc:237] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:34.097287 11937 consensus_queue.cc:1046] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20240426 09:55:34.111065 11968 consensus_queue.cc:237] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:34.137066 11978 consensus_queue.cc:1035] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.124s
I20240426 09:55:34.176087 11964 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:34.162739 11857 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:34.159050 11858 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:34.163354 11829 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:34.188861 10012 raft_consensus.cc:2379] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
W20240426 09:55:34.179411 11989 log.cc:924] Time spent T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.161s user 0.001s sys 0.000s
I20240426 09:55:34.158670 11945 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:34.172801 11733 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:34.206318 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:34.208194 9932 raft_consensus.cc:2379] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:34.208807 9937 raft_consensus.cc:1261] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20240426 09:55:34.204015 11991 log.cc:924] Time spent T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.159s user 0.000s sys 0.001s
I20240426 09:55:34.191555 11858 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:34.189288 11877 raft_consensus.cc:686] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:34.162837 11980 consensus_queue.cc:1035] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.152s
I20240426 09:55:34.237033 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:34.239177 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:34.240289 11868 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:34.240845 10056 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:34.242969 10056 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:34.250793 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:34.257540 9932 raft_consensus.cc:2379] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:34.262308 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:34.262971 9937 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:34.271943 10056 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:34.261512 11829 raft_consensus.cc:2726] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:34.271688 11996 raft_consensus.cc:2781] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:34.349761 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:34.350244 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:34.351894 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:34.305945 11955 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:34.308749 10056 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:34.278746 11994 raft_consensus.cc:2781] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election won for term 5
W20240426 09:55:34.352137 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 320503us
W20240426 09:55:34.388422 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 79716.3us
I20240426 09:55:34.359601 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:34.391539 10008 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:34.309722 11877 consensus_queue.cc:237] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:34.395073 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:34.370679 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:34.375687 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:34.391978 9979 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
W20240426 09:55:34.348762 11967 consensus_meta.cc:287] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.738s user 0.000s sys 0.009s
I20240426 09:55:34.360908 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:34.396322 9932 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
W20240426 09:55:34.379156 12000 log.cc:924] Time spent T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20240426 09:55:34.399901 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:34.310346 11996 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 09:55:34.426983 12002 log.cc:924] Time spent T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20240426 09:55:34.391146 11995 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:34.377605 11999 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
W20240426 09:55:34.431080 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 399131us
W20240426 09:55:34.440536 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 48642.8us
I20240426 09:55:34.441700 9979 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:34.324055 11804 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:34.432432 9934 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:34.432785 10008 raft_consensus.cc:2379] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 6: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:34.392453 11769 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:55:34.360224 11947 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:34.370527 11944 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:34.446796 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:34.398296 11994 raft_consensus.cc:686] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:34.446638 11947 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:34.311134 11830 consensus_queue.cc:1035] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.089s
W20240426 09:55:34.439263 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 46537.6us
I20240426 09:55:34.451957 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:34.439129 11893 consensus_meta.cc:287] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.527s user 0.007s sys 0.002s
I20240426 09:55:34.454625 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:34.438908 12005 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:34.462972 12011 raft_consensus.cc:2726] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 09:55:34.466228 12003 log.cc:924] Time spent T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.118s user 0.001s sys 0.000s
W20240426 09:55:34.466329 11875 consensus_meta.cc:287] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.566s user 0.009s sys 0.000s
I20240426 09:55:34.466354 11994 consensus_queue.cc:237] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:34.468155 11893 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:34.442232 11995 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:34.443360 11999 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:34.490497 9982 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:34.491379 9982 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:34.495371 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:34.495939 11644 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:34.496410 11644 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:34.467073 11944 consensus_queue.cc:1046] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20240426 09:55:34.467082 11947 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:34.506636 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:34.507277 10084 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:34.513587 10055 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
W20240426 09:55:34.516999 11985 log.cc:924] Time spent T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.799s user 0.002s sys 0.000s
I20240426 09:55:34.517482 11502 raft_consensus.cc:3032] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:34.517838 11502 raft_consensus.cc:729] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:34.518008 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:34.518569 11981 log.cc:924] Time spent T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.826s user 0.000s sys 0.001s
I20240426 09:55:34.519730 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1431) took 1023 ms. Trace:
I20240426 09:55:34.520077 10010 rpcz_store.cc:276] 0426 09:55:33.496363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:33.496783 (+ 420us) service_pool.cc:225] Handling call
0426 09:55:33.520374 (+ 23591us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:33.522735 (+ 2361us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:33.522784 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:33.537892 (+ 15108us) log.cc:842] Serialized 26 byte log entry
0426 09:55:33.640098 (+102206us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:33.642228 (+ 2130us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:33.642288 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:34.515841 (+873553us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:55:34.519440 (+ 3599us) raft_consensus.cc:1671] finished
0426 09:55:34.519457 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:34.519679 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2003,"raft.queue_time_us":2600,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":10524,"spinlock_wait_cycles":27073408,"thread_start_us":71616,"threads_started":1,"wal-append.queue_time_us":167924}
I20240426 09:55:34.521932 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:34.524412 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:34.464699 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:34.525549 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:34.526393 12006 log.cc:924] Time spent T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.151s user 0.000s sys 0.000s
I20240426 09:55:34.526527 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:34.531231 11502 consensus_queue.cc:260] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:34.532023 11997 log.cc:924] Time spent T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.174s user 0.000s sys 0.001s
W20240426 09:55:34.533604 12008 log.cc:924] Time spent T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.160s user 0.001s sys 0.000s
I20240426 09:55:34.536068 10014 raft_consensus.cc:1261] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:34.511137 11230 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:34.539278 11230 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:34.539615 11230 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:34.540300 11958 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:34.541574 10054 leader_election.cc:398] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Vote denied by peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313) with higher term. Message: Invalid argument: T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4.
I20240426 09:55:34.541885 10054 leader_election.cc:401] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20240426 09:55:34.544495 9932 raft_consensus.cc:1261] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240426 09:55:34.447870 11996 raft_consensus.cc:3037] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:34.484247 10008 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:34.418694 11967 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:34.447669 11769 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:34.459818 11733 raft_consensus.cc:2726] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:34.487282 9979 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:34.552501 10008 raft_consensus.cc:595] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:34.450843 11804 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:34.460434 10618 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:34.489158 11875 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:34.491786 12005 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:34.748219 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 201960us
W20240426 09:55:34.751988 9906 leader_election.cc:334] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): Timed out: RequestConsensusVote RPC to 127.8.141.130:44411 timed out after 1.556s (SENT)
I20240426 09:55:34.757115 9906 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:34.581923 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 94704.9us
W20240426 09:55:34.581491 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 90178.7us
W20240426 09:55:34.609865 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 81413.9us
I20240426 09:55:34.620414 10015 raft_consensus.cc:2379] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:34.789837 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:34.790632 10014 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
W20240426 09:55:34.764914 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 251395us
I20240426 09:55:34.791898 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:34.791975 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:34.792627 10088 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:34.793177 10014 raft_consensus.cc:2379] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:34.776293 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:34.795882 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 58752.3us
I20240426 09:55:34.796177 10090 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 5.
I20240426 09:55:34.797118 9906 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:34.800021 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:34.800232 10583 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:34.801401 10085 raft_consensus.cc:2379] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 5: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
W20240426 09:55:34.793551 10055 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.141.130:44411 timed out after 2.379s (SENT) [suppressed 8 similar messages]
W20240426 09:55:34.803162 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 46047.8us
I20240426 09:55:34.579108 11769 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:34.763254 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:34.609439 11945 raft_consensus.cc:2726] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 09:55:34.794751 11972 consensus_meta.cc:287] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.747s user 0.007s sys 0.003s
I20240426 09:55:34.827375 11972 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:34.828609 9941 raft_consensus.cc:3037] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:34.810931 12014 raft_consensus.cc:2726] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:34.610400 11967 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:34.609177 12011 raft_consensus.cc:2781] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:34.668383 11893 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:34.748121 11999 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:34.747591 10616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45564 (request call id 244) took 1387 ms. Trace:
I20240426 09:55:34.811216 11814 raft_consensus.cc:2726] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:34.817910 11875 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:34.821024 10009 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1399) took 2641 ms (client timeout 2379 ms). Trace:
I20240426 09:55:34.822018 10016 raft_consensus.cc:1261] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20240426 09:55:34.822806 11955 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:34.823514 10083 raft_consensus.cc:1261] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 09:55:34.832623 11804 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:34.809108 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:34.834105 11968 raft_consensus.cc:2726] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:55:34.834599 10015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1437) took 1121 ms. Trace:
I20240426 09:55:34.805655 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:34.670117 11995 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:34.797118 11462 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:34.646800 11858 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:34.646740 11644 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:35.235090 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 441524us
W20240426 09:55:35.228641 10013 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.005s sys 0.014s
W20240426 09:55:35.371438 10013 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s user 0.006s sys 0.018s
I20240426 09:55:34.861820 10012 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1434) took 1345 ms. Trace:
I20240426 09:55:34.803303 11944 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:35.304996 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 507900us
I20240426 09:55:35.385480 9906 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:34.995884 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 1095) took 1148 ms. Trace:
W20240426 09:55:35.385607 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 408800us
I20240426 09:55:34.804461 11679 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election won for term 4
W20240426 09:55:35.258301 10010 consensus_meta.cc:287] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.736s user 0.010s sys 0.001s
I20240426 09:55:34.862390 10082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1199) took 1160 ms. Trace:
I20240426 09:55:35.388936 10082 rpcz_store.cc:276] 0426 09:55:33.701632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:33.701776 (+ 144us) service_pool.cc:225] Handling call
0426 09:55:34.862208 (+1160432us) spinlock_profiling.cc:243] Waited 6.85 ms on lock 0x7b54000daf54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:34.862333 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15745280}
I20240426 09:55:35.381131 10015 rpcz_store.cc:276] 0426 09:55:33.713363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:33.713554 (+ 191us) service_pool.cc:225] Handling call
0426 09:55:34.834438 (+1120884us) spinlock_profiling.cc:243] Waited 892 ms on lock 0x7b540016ce54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:34.834556 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2052088064}
I20240426 09:55:35.372630 10012 rpcz_store.cc:276] 0426 09:55:33.516417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:33.516550 (+ 133us) service_pool.cc:225] Handling call
0426 09:55:34.861607 (+1345057us) spinlock_profiling.cc:243] Waited 636 ms on lock 0x7b54002ffdd4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:34.861757 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1462485888}
I20240426 09:55:34.992826 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1208) took 1255 ms. Trace:
I20240426 09:55:35.396737 10087 rpcz_store.cc:276] 0426 09:55:33.737082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:33.737187 (+ 105us) service_pool.cc:225] Handling call
0426 09:55:34.992606 (+1255419us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b540037e454. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:34.992762 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":704270720}
I20240426 09:55:34.837687 9941 raft_consensus.cc:2454] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:34.907790 10088 raft_consensus.cc:2454] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:35.376719 10016 pending_rounds.cc:76] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting all ops after (but not including) 2
W20240426 09:55:35.379392 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 497306us
I20240426 09:55:35.397965 10016 pending_rounds.cc:97] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting uncommitted NO_OP operation due to leader change: 4.3
W20240426 09:55:35.382790 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 568142us
I20240426 09:55:35.386272 10007 rpcz_store.cc:276] 0426 09:55:33.847310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:34.046182 (+198872us) service_pool.cc:225] Handling call
0426 09:55:34.995717 (+949535us) spinlock_profiling.cc:243] Waited 75.5 ms on lock 0x7b540030f7d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:34.995826 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":173748096}
W20240426 09:55:35.377254 10009 rpcz_store.cc:269] 0426 09:55:32.179321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:32.189426 (+ 10105us) service_pool.cc:225] Handling call
0426 09:55:34.820821 (+2631395us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540030f2d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:34.820959 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2685592448}
I20240426 09:55:35.398268 10016 raft_consensus.cc:2866] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:35.387871 10010 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:55:35.423071 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:35.425873 10014 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1444) took 1504 ms. Trace:
I20240426 09:55:35.426334 10014 rpcz_store.cc:276] 0426 09:55:33.921942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:35.425035 (+1503093us) service_pool.cc:225] Handling call
0426 09:55:35.425404 (+ 369us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:35.425576 (+ 172us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:35.425598 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:35.425636 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:35.425665 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:35.425716 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:35.425825 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:34.807682 11948 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:34.860332 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:34.838225 11951 consensus_queue.cc:1035] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.696s
I20240426 09:55:35.819224 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:35.821650 10014 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 259) took 1053 ms. Trace:
I20240426 09:55:35.480373 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:35.560772 10620 raft_consensus.cc:1261] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20240426 09:55:35.832967 9933 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.962s user 0.016s sys 0.051s
I20240426 09:55:35.464340 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:35.465334 10007 raft_consensus.cc:1261] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20240426 09:55:35.833384 9933 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.962s user 0.016s sys 0.051s
I20240426 09:55:35.833725 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1445) took 1828 ms. Trace:
I20240426 09:55:35.185761 11696 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:35.186164 11877 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.057s sys 0.105s
I20240426 09:55:35.464012 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:35.820046 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 254) took 1649 ms. Trace:
W20240426 09:55:35.821825 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 436426us
I20240426 09:55:35.540040 10015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1112) took 1618 ms. Trace:
I20240426 09:55:35.820770 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 258) took 1052 ms. Trace:
I20240426 09:55:35.832849 10087 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:35.656739 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1189) took 1281 ms. Trace:
I20240426 09:55:35.821254 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:35.373452 12011 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:34.873324 11976 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:34.874413 11958 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:35.834172 10007 rpcz_store.cc:276] 0426 09:55:34.005704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:35.428303 (+1422599us) service_pool.cc:225] Handling call
0426 09:55:35.428844 (+ 541us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:35.833500 (+404656us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:35.833669 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:34.843091 11972 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:35.501685 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 696498us
W20240426 09:55:35.811278 12020 log.cc:924] Time spent T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.309s user 0.000s sys 0.000s
I20240426 09:55:35.822909 11807 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:35.837568 9906 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:35.898633 10015 rpcz_store.cc:276] 0426 09:55:33.922101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:35.427562 (+1505461us) service_pool.cc:225] Handling call
0426 09:55:35.539346 (+111784us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:35.539655 (+ 309us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:35.539698 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:35.539761 (+ 63us) raft_consensus.cc:1631] Marking committed up to 4
0426 09:55:35.539790 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:35.539861 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:35.539982 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:35.899451 9935 rpcz_store.cc:276] 0426 09:55:34.375301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:34.393176 (+ 17875us) service_pool.cc:225] Handling call
0426 09:55:34.393471 (+ 295us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:34.393759 (+ 288us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:34.393787 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:34.397185 (+ 3398us) log.cc:842] Serialized 26 byte log entry
0426 09:55:34.818269 (+421084us) spinlock_profiling.cc:243] Waited 1.08 ms on lock 0x7b180032cc20. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fcfdb75 00007f9ca8b37789 00007f9c9fd1b0eb 00007f9c9fd194ef 00007f9c9fd1b652 00007f9ca28c743b 00007f9ca28ccee3 00007f9ca28cd291 00007f9ca2c1a75f
0426 09:55:35.219004 (+400735us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:35.219047 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:35.219120 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:35.656428 (+437308us) raft_consensus.cc:1671] finished
0426 09:55:35.656450 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:35.656675 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":246,"raft.queue_time_us":3130,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":1560,"spinlock_wait_cycles":2476416,"thread_start_us":400546,"threads_started":1,"wal-append.queue_time_us":1257669}
I20240426 09:55:34.860720 10084 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
I20240426 09:55:35.838302 11980 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:35.897697 11771 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:35.910177 11688 raft_consensus.cc:2726] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313) with higher term. Message: Invalid argument: T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4.
I20240426 09:55:35.824433 11936 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:35.640859 11875 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:35.910964 11807 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:35.911360 11875 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:35.911979 11893 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:35.912420 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:35.835312 11417 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:35.910286 10611 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51158 (request call id 196) took 1077 ms. Trace:
I20240426 09:55:35.900856 11967 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:35.428057 11944 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:35.912492 11893 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:35.917068 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:35.917213 11644 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:35.904706 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:35.918301 10082 raft_consensus.cc:2379] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 4: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
W20240426 09:55:35.838564 11877 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.195s user 0.071s sys 0.138s
W20240426 09:55:35.904872 9904 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.097s user 0.011s sys 0.013s
I20240426 09:55:35.905792 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
W20240426 09:55:35.579833 11230 consensus_meta.cc:287] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 1.040s user 0.009s sys 0.000s
I20240426 09:55:35.068990 11996 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:35.838450 10010 rpcz_store.cc:276] 0426 09:55:34.767922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:35.426686 (+658764us) service_pool.cc:225] Handling call
0426 09:55:35.820308 (+393622us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:35.820480 (+ 172us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:35.820504 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:35.820543 (+ 39us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:55:35.820570 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:35.820619 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:35.820722 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:35.565645 11995 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:35.909327 10055 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:35.540879 10012 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1115) took 1564 ms. Trace:
I20240426 09:55:35.910594 11771 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:35.838387 11943 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:35.914602 11967 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:35.836370 10009 rpcz_store.cc:276] 0426 09:55:34.170281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:35.429425 (+1259144us) service_pool.cc:225] Handling call
0426 09:55:35.429907 (+ 482us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:35.430121 (+ 214us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:35.430146 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:35.430185 (+ 39us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:55:35.430212 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:35.430262 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:35.819987 (+389725us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:35.774917 11774 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:35.667584 12005 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:35.621603 11809 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:35.825088 11938 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.953s user 0.011s sys 0.035s
I20240426 09:55:35.982831 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34362 (request call id 734) took 2079 ms. Trace:
W20240426 09:55:35.921643 11994 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.902s user 0.032s sys 0.040s
I20240426 09:55:35.983889 10009 rpcz_store.cc:276] 0426 09:55:33.903690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:35.981821 (+2078131us) service_pool.cc:225] Handling call
0426 09:55:35.982755 (+ 934us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:35.903144 12011 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:35.621773 11804 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:55:35.901499 9937 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.037s user 0.003s sys 0.020s
I20240426 09:55:35.822115 10014 rpcz_store.cc:276] 0426 09:55:34.768149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:35.427459 (+659310us) service_pool.cc:225] Handling call
0426 09:55:35.821212 (+393753us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:35.821366 (+ 154us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:35.821389 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:35.821426 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:55:35.821464 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:35.821509 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:35.821601 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:35.593685 11782 raft_consensus.cc:2726] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 09:55:35.920384 9904 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.113s user 0.011s sys 0.014s
W20240426 09:55:35.920892 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 83392.3us
I20240426 09:55:35.173337 11769 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:35.914069 11805 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:55:35.949502 10008 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.152s user 0.008s sys 0.041s
I20240426 09:55:35.424602 11955 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:35.839375 11741 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:35.976871 10012 rpcz_store.cc:276] 0426 09:55:33.976722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:35.437377 (+1460655us) service_pool.cc:225] Handling call
0426 09:55:35.540440 (+103063us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:35.540603 (+ 163us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:35.540626 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:35.540663 (+ 37us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:55:35.540690 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:35.540745 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:35.540830 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:35.979611 10015 raft_consensus.cc:1226] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 5. Current term is 6. Ops: []
I20240426 09:55:35.412847 11462 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:35.980940 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 71694.4us
I20240426 09:55:35.982004 10010 raft_consensus.cc:1261] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 09:55:35.982528 10090 raft_consensus.cc:2379] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 4: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
W20240426 09:55:35.847779 9979 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.034s user 0.009s sys 0.011s
W20240426 09:55:35.985749 9937 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.121s user 0.004s sys 0.020s
I20240426 09:55:35.831077 11948 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:35.984375 10088 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:35.988057 9906 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:36.004698 9906 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:35.928175 11810 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.083s user 0.007s sys 0.019s
I20240426 09:55:35.912971 11807 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:35.923154 10084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 182) took 1414 ms. Trace:
I20240426 09:55:35.910987 11936 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:35.913226 11875 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:35.975595 11230 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:35.985045 11938 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.113s user 0.012s sys 0.040s
I20240426 09:55:35.974206 12031 raft_consensus.cc:2726] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:35.982987 11995 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:35.986102 11994 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.966s user 0.032s sys 0.041s
I20240426 09:55:35.993548 10015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1453) took 1181 ms. Trace:
I20240426 09:55:36.019632 10015 rpcz_store.cc:276] 0426 09:55:34.811894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:35.909126 (+1097232us) service_pool.cc:225] Handling call
0426 09:55:35.979492 (+ 70366us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:35.993318 (+ 13826us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:35.993500 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:55:36.010581 11810 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.195s user 0.009s sys 0.021s
I20240426 09:55:35.997295 11771 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:35.997169 11462 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:35.997817 11741 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:35.914258 10611 rpcz_store.cc:276] 0426 09:55:34.833162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:34.835001 (+ 1839us) service_pool.cc:225] Handling call
0426 09:55:35.910225 (+1075224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:35.974249 11644 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20240426 09:55:35.994912 11805 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:35.919106 11958 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:35.999929 11502 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:35.388025 10616 rpcz_store.cc:276] 0426 09:55:33.360556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:33.370025 (+ 9469us) service_pool.cc:225] Handling call
0426 09:55:33.493661 (+123636us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:33.567244 (+ 73583us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:34.747297 (+1180053us) spinlock_profiling.cc:243] Waited 80.8 ms on lock 0x7b54005055d0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c45b99 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:34.747484 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":185882880}
I20240426 09:55:36.036015 11936 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:35.897295 11696 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.007952 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:35.913108 11980 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:35.566740 11999 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:35.913688 11417 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:35.990329 10008 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.193s user 0.010s sys 0.043s
I20240426 09:55:35.991341 11979 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:35.979275 11774 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.009310 11782 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:35.981581 11943 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.000010 11769 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:35.921828 11944 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:35.986308 11809 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:35.995540 9979 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.181s user 0.012s sys 0.012s
I20240426 09:55:35.994446 10055 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:35.986820 11804 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.008819 11972 raft_consensus.cc:2726] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 09:55:36.052778 10013 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.016s sys 0.033s
I20240426 09:55:36.007941 11947 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:36.013402 11951 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:36.016791 11807 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:36.024300 11810 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:36.019448 11875 raft_consensus.cc:2781] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:36.012327 10084 rpcz_store.cc:276] 0426 09:55:34.506217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:34.506345 (+ 128us) service_pool.cc:225] Handling call
0426 09:55:35.920540 (+1414195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:55:36.039901 11771 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:36.042045 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:36.044464 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1105) took 2194 ms. Trace:
I20240426 09:55:36.027848 11967 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:36.036407 11965 raft_consensus.cc:2726] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:36.042519 11999 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:36.051658 10056 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.019s sys 0.029s
W20240426 09:55:36.074435 10056 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.572s user 0.020s sys 0.030s
W20240426 09:55:36.054582 10013 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.591s user 0.017s sys 0.033s
W20240426 09:55:36.054380 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 60016us
I20240426 09:55:36.053881 11769 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:36.060374 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:36.081900 10014 raft_consensus.cc:2454] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:36.043787 11980 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:36.044049 11995 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:36.042367 11741 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:36.063977 10081 raft_consensus.cc:2454] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 2.
I20240426 09:55:36.066673 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:36.066527 11937 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:36.054847 11679 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.015s sys 0.034s
I20240426 09:55:36.068569 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:36.077621 10055 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:36.068368 11875 raft_consensus.cc:686] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:36.055737 12037 raft_consensus.cc:2726] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:55:36.065035 11996 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:36.063711 11893 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:36.068360 12029 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:36.089013 11679 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.550s user 0.015s sys 0.035s
I20240426 09:55:36.082154 11877 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:36.082468 11938 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:36.064425 11951 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.069530 10008 rpcz_store.cc:276] 0426 09:55:33.849563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:34.610841 (+761278us) service_pool.cc:225] Handling call
0426 09:55:34.611158 (+ 317us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:34.611352 (+ 194us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 09:55:34.797629 (+186277us) log.cc:842] Serialized 14 byte log entry
0426 09:55:36.044159 (+1246530us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:36.044210 (+ 51us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:36.044249 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:36.044310 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:36.044416 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1274,"prepare.queue_time_us":921986,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":1346,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":4280,"thread_start_us":1431390,"threads_started":2}
I20240426 09:55:36.059674 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:36.051295 11782 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.090468 10013 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1102) took 2241 ms. Trace:
I20240426 09:55:36.087905 11948 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:36.051661 11979 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:36.062803 11947 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.079725 11994 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:36.100931 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1451) took 1308 ms. Trace:
I20240426 09:55:36.079663 11462 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:36.110790 10012 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:36.067585 11810 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.083591 11955 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:36.108359 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 205373us
I20240426 09:55:36.084638 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:36.088789 11967 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:36.090507 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:36.141871 12010 log.cc:924] Time spent T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 1.714s user 0.002s sys 0.008s
I20240426 09:55:36.075599 11771 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:36.143579 12026 log.cc:924] Time spent T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20240426 09:55:36.143855 9941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 1188) took 1782 ms (client timeout 1684 ms). Trace:
I20240426 09:55:36.144199 11771 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:36.079488 11976 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:36.108747 10013 rpcz_store.cc:276] 0426 09:55:33.848867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:34.545355 (+696488us) service_pool.cc:225] Handling call
0426 09:55:34.579497 (+ 34142us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:34.579828 (+ 331us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:34.579860 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:34.580266 (+ 406us) log.cc:842] Serialized 26 byte log entry
0426 09:55:35.371825 (+791559us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:55:36.079299 (+707474us) log.cc:842] Serialized 14 byte log entry
0426 09:55:36.090184 (+ 10885us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:36.090254 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:36.090293 (+ 39us) raft_consensus.cc:1671] finished
0426 09:55:36.090303 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:36.090420 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14307,"prepare.queue_time_us":205517,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"raft.queue_time_us":12331,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":46692,"thread_start_us":1327673,"threads_started":2,"wal-append.queue_time_us":1157584}
I20240426 09:55:36.065306 11807 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:36.085232 11769 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:36.105716 11679 raft_consensus.cc:686] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
W20240426 09:55:36.104187 12027 log.cc:924] Time spent T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.098s user 0.000s sys 0.001s
I20240426 09:55:36.135449 10007 rpcz_store.cc:276] 0426 09:55:34.792387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:35.907602 (+1115215us) service_pool.cc:225] Handling call
0426 09:55:35.907897 (+ 295us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:35.908093 (+ 196us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 09:55:35.914325 (+ 6232us) log.cc:842] Serialized 14 byte log entry
0426 09:55:36.100644 (+186319us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:36.100691 (+ 47us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:55:36.100727 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:36.100783 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:36.100883 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11132,"prepare.queue_time_us":5452,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":11319,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":3805,"thread_start_us":175265,"threads_started":1}
W20240426 09:55:36.144254 9941 rpcz_store.cc:269] 0426 09:55:34.361601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:34.375449 (+ 13848us) service_pool.cc:225] Handling call
0426 09:55:36.143708 (+1768259us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b54002373d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:36.143795 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":994022656}
I20240426 09:55:36.135789 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:36.160696 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1165) took 2440 ms. Trace:
I20240426 09:55:36.110285 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
W20240426 09:55:36.163223 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 85686.2us
I20240426 09:55:36.164291 9933 rpcz_store.cc:276] 0426 09:55:33.720126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:33.722374 (+ 2248us) service_pool.cc:225] Handling call
0426 09:55:33.746023 (+ 23649us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:33.761760 (+ 15737us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:33.761788 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:33.762166 (+ 378us) log.cc:842] Serialized 26 byte log entry
0426 09:55:34.871008 (+1108842us) spinlock_profiling.cc:243] Waited 63.1 ms on lock 0x7b180040f300. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fcfdb75 00007f9ca8b37789 00007f9c9fd1b0eb 00007f9c9fd194ef 00007f9c9fd1b652 00007f9ca28c743b 00007f9ca28ccee3 00007f9ca28cd291 00007f9ca2c1a75f
0426 09:55:35.898279 (+1027271us) raft_consensus.cc:1631] Marking committed up to 1
0426 09:55:35.898314 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:35.898375 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:36.144886 (+246511us) raft_consensus.cc:1671] finished
0426 09:55:36.144904 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:36.160637 (+ 15733us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2505,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":6689,"spinlock_wait_cycles":145037568,"thread_start_us":1027076,"threads_started":1,"wal-append.queue_time_us":2297844}
I20240426 09:55:36.133589 11948 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
W20240426 09:55:36.167374 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 188422us
I20240426 09:55:36.188171 10055 leader_election.cc:398] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4.
I20240426 09:55:36.108464 11733 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20240426 09:55:36.189633 10055 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:36.168653 12005 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:36.137570 11230 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:36.188410 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 183772us
I20240426 09:55:36.152269 12039 raft_consensus.cc:2726] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:36.221678 12035 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:36.107089 12029 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.222445 11679 consensus_queue.cc:237] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.224146 11804 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:36.241894 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 52321.6us
I20240426 09:55:36.226300 11951 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:36.160199 11976 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.226986 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:36.091399 11937 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.136873 11994 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.168490 11696 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:36.228334 11417 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:36.109601 11877 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.187816 11875 consensus_queue.cc:237] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.231716 12005 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.233356 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:36.235396 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1104) took 2386 ms. Trace:
I20240426 09:55:36.290426 10011 rpcz_store.cc:276] 0426 09:55:33.849331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:34.551111 (+701780us) service_pool.cc:225] Handling call
0426 09:55:34.581602 (+ 30491us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:34.621376 (+ 39774us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:34.621407 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:34.621449 (+ 42us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:34.621477 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:36.235185 (+1613708us) spinlock_profiling.cc:243] Waited 39.6 ms on lock 0x7b54000573d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:55:36.235233 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:36.235345 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":91007232}
I20240426 09:55:36.109155 11938 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:36.162050 11805 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:55:36.239091 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 205522us
I20240426 09:55:36.239157 9906 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:36.150986 11771 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:36.224795 12036 log.cc:924] Time spent T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20240426 09:55:36.243357 11893 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:36.246518 11721 raft_consensus.cc:3032] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:36.150413 11967 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:36.248678 11999 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:36.241925 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:36.242172 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:36.245853 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:36.246413 10012 raft_consensus.cc:595] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:36.158903 11462 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:36.247915 10055 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:36.169458 11958 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:36.279745 11502 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:36.188632 11948 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.322261 10012 raft_consensus.cc:603] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240426 09:55:36.166405 12028 log.cc:924] Time spent T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.060s user 0.000s sys 0.001s
I20240426 09:55:36.248782 11943 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:36.309932 12024 log.cc:924] Time spent T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.210s user 0.001s sys 0.000s
I20240426 09:55:36.312281 9938 raft_consensus.cc:1226] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 5. Current term is 6. Ops: []
I20240426 09:55:36.245767 12040 raft_consensus.cc:2726] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:36.328188 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 1407) took 1843 ms. Trace:
I20240426 09:55:36.322124 9939 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
W20240426 09:55:36.331869 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 93436.3us
I20240426 09:55:36.248656 11809 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:36.333182 9906 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:36.334053 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:36.334004 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:36.335445 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:36.332599 9937 rpcz_store.cc:276] 0426 09:55:34.484931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:34.485061 (+ 130us) service_pool.cc:225] Handling call
0426 09:55:34.485385 (+ 324us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 09:55:34.485931 (+ 546us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:34.485958 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 09:55:34.486443 (+ 485us) log.cc:842] Serialized 52 byte log entry
0426 09:55:34.864621 (+378178us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b180037ba00. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fcfdb75 00007f9ca8b37789 00007f9c9fd1b0eb 00007f9c9fd194ef 00007f9c9fd1b652 00007f9ca28c743b 00007f9ca28ccee3 00007f9ca28cd291 00007f9ca2c1a75f
0426 09:55:35.995191 (+1130570us) raft_consensus.cc:1631] Marking committed up to 2
0426 09:55:35.995230 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:35.995318 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:36.327929 (+332611us) raft_consensus.cc:1671] finished
0426 09:55:36.327947 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:36.328136 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":229,"raft.queue_time_us":308,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":5656,"spinlock_wait_cycles":30835968,"thread_start_us":1130386,"threads_started":1,"wal-append.queue_time_us":1583077}
I20240426 09:55:36.334034 9906 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:36.336606 9935 raft_consensus.cc:2396] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20240426 09:55:36.337819 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:36.337217 9906 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:36.337284 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:36.336014 9941 raft_consensus.cc:2454] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 1.
I20240426 09:55:36.341570 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:36.342844 9906 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:36.346082 9939 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
W20240426 09:55:36.346339 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 98504.7us
I20240426 09:55:36.349489 11462 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:36.312196 11957 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:36.354144 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:36.351668 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:36.356103 12047 consensus_queue.cc:1035] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:36.322378 11721 raft_consensus.cc:729] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:36.360673 12032 raft_consensus.cc:2726] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:36.346002 11951 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:36.363004 12005 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:36.364409 11937 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:36.358572 12044 log.cc:924] Time spent T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20240426 09:55:36.357820 11807 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:36.383422 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:36.396334 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:36.380668 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:36.396576 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:36.431377 9933 raft_consensus.cc:2379] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:36.309988 12045 consensus_queue.cc:1035] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.044s
I20240426 09:55:36.316488 11771 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.407433 10616 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:36.409632 12029 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:36.326360 11948 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:36.328169 11502 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.442438 11502 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:36.414654 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 66503.9us
W20240426 09:55:36.415201 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 59768.9us
I20240426 09:55:36.416605 11956 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:36.425175 11804 raft_consensus.cc:2726] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:36.365954 12039 raft_consensus.cc:2781] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:55:36.316356 11893 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.353039 11805 raft_consensus.cc:2741] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20240426 09:55:36.481138 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:36.481715 10011 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:36.482080 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:36.366865 11943 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:36.323866 12048 raft_consensus.cc:2726] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:36.440567 11771 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:36.451161 10054 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
W20240426 09:55:36.452133 12042 log.cc:924] Time spent T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20240426 09:55:36.452672 11810 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:36.453379 11938 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:55:36.435948 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 139273us
I20240426 09:55:36.454250 11976 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:36.454811 11769 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:36.455895 11877 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:36.456936 11967 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:36.457399 11721 consensus_queue.cc:260] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.412163 11952 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:36.457999 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:36.460063 11994 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:36.438239 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:36.352795 11809 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:36.423705 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:36.474218 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:36.476790 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:36.477324 9979 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:36.411568 11957 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.244019 12035 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.481016 11733 consensus_queue.cc:1046] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20240426 09:55:36.425918 11951 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:36.504500 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 53427.3us
I20240426 09:55:36.504822 10583 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:36.505637 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:36.507700 9939 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:36.472973 11937 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:36.477298 11829 consensus_queue.cc:1035] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20240426 09:55:36.476022 11996 consensus_queue.cc:1046] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20240426 09:55:36.476406 11956 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.510644 9937 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
W20240426 09:55:36.509620 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 94910.8us
I20240426 09:55:36.679589 10011 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:36.678182 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 221) took 2702 ms. Trace:
W20240426 09:55:36.681262 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 176525us
I20240426 09:55:36.556871 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:36.683454 10014 raft_consensus.cc:2379] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 6: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:36.684322 10013 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
W20240426 09:55:36.624907 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 158689us
I20240426 09:55:36.682155 9936 rpcz_store.cc:276] 0426 09:55:33.975995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:33.976126 (+ 131us) service_pool.cc:225] Handling call
0426 09:55:33.976409 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:33.976583 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 09:55:33.998962 (+ 22379us) log.cc:842] Serialized 14 byte log entry
0426 09:55:34.007153 (+ 8191us) log.cc:842] Serialized 14 byte log entry
0426 09:55:34.534056 (+526903us) spinlock_profiling.cc:243] Waited 3.57 ms on lock 0x7b18000f4360. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fd1bfa4 00007f9c9fd1f351 00007f9c9fd1f309 00007f9c9fd1f299 00007f9c9fd1f261 00007f9c9fd1e55d 00007f9ca8f8fd34 00007f9ca8f8fb69 00007f9c9fcfe5f6
0426 09:55:34.861065 (+327009us) spinlock_profiling.cc:243] Waited 3.18 ms on lock 0x7b480030b658. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca7aee2e1 00007f9ca7ae53d7 00007f9ca7ae526a 00007f9ca7bd943f 00007f9ca7bd9107 00007f9ca7bed60c 00007f9ca7bed5b8 00007f9ca7bed538 00007f9ca7bed4f3 00007f9ca7bec79c 00007f9ca6b6f8d6
0426 09:55:36.473066 (+1612001us) spinlock_profiling.cc:243] Waited 7.8 ms on lock 0x7b18000f4360. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fc94faa 00007f9ca9383f25 00007f9ca28d74e3 00007f9ca28cceaa 00007f9ca28cd3fb 00007f9ca2c50a1a 00007f9ca2c6009f 00007f9ca2c60018 00007f9ca2c5ff98
0426 09:55:36.670140 (+197074us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:55:36.670202 (+ 62us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:55:36.670245 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:36.677986 (+ 7741us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:36.678122 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28055,"prepare.queue_time_us":70848,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":61,"raft.queue_time_us":163275,"raft.run_cpu_time_us":11682,"raft.run_wall_time_us":871780,"spinlock_wait_cycles":33440256,"thread_start_us":191951,"threads_started":1}
I20240426 09:55:36.499356 12040 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:36.687225 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 209960us
I20240426 09:55:36.688759 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:36.496904 12039 raft_consensus.cc:686] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:36.689344 10086 raft_consensus.cc:1226] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:36.690025 9906 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:36.678826 12011 consensus_meta.cc:287] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.687s user 0.010s sys 0.000s
I20240426 09:55:36.692219 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:36.693022 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:36.696089 10088 raft_consensus.cc:2396] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 09:55:36.696655 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:36.696856 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:36.697733 10088 raft_consensus.cc:2396] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240426 09:55:36.508478 11771 raft_consensus.cc:2781] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 09:55:36.691130 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 287721us
I20240426 09:55:36.694381 9979 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:36.694721 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:36.698925 10013 raft_consensus.cc:2379] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:36.694283 9935 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:36.700071 9979 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:36.701790 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:36.704273 9904 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:36.705667 9979 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:36.707051 10088 raft_consensus.cc:1226] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 6. Current term is 7. Ops: []
I20240426 09:55:36.678580 11955 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:36.710714 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:36.503813 11943 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.712684 10083 raft_consensus.cc:1261] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:36.670375 11814 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:36.689018 11937 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.688068 11952 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:36.687350 11960 raft_consensus.cc:2781] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:36.970739 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:36.810525 9935 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:55:36.840590 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:36.855255 10085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59762 (request call id 1062) took 3133 ms (client timeout 2084 ms). Trace:
I20240426 09:55:36.730855 11733 raft_consensus.cc:2781] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:55:36.830634 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:36.731894 10081 raft_consensus.cc:1226] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20240426 09:55:36.990262 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:36.840215 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:36.987416 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 254053us
I20240426 09:55:36.989552 10013 raft_consensus.cc:2379] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:36.697834 11721 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:36.691799 11778 raft_consensus.cc:2726] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:55:36.985584 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:36.992038 9906 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:36.779100 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:36.780346 11957 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:36.994491 11814 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:36.992549 10085 rpcz_store.cc:269] 0426 09:55:33.721567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:33.721957 (+ 390us) service_pool.cc:225] Handling call
0426 09:55:36.855034 (+3133077us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400085054. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:36.855151 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2466594560}
I20240426 09:55:36.997596 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
W20240426 09:55:36.778702 12060 log.cc:924] Time spent T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.182s user 0.001s sys 0.000s
I20240426 09:55:36.819592 12039 consensus_queue.cc:237] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:36.888373 11858 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:36.778268 12065 log.cc:924] Time spent T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20240426 09:55:36.704926 11771 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:36.709266 12005 raft_consensus.cc:2726] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:37.003086 9906 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:36.851382 12062 log.cc:924] Time spent T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20240426 09:55:36.997548 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:37.003099 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:37.096377 10088 raft_consensus.cc:2379] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 4: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:37.097191 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.097803 10088 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
W20240426 09:55:37.015259 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 245687us
I20240426 09:55:37.024823 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.007615 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.007911 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:36.996831 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:37.030050 10082 raft_consensus.cc:2379] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 4: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:55:37.030889 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:37.024354 11937 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:36.767242 11644 raft_consensus.cc:2781] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:36.820482 11691 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:36.779415 12063 raft_consensus.cc:2781] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:36.976383 11809 raft_consensus.cc:2726] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:36.889803 11774 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:37.033751 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.001915 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:36.780740 11893 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:36.819765 12035 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:36.750454 11955 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:36.998590 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:36.731606 11230 raft_consensus.cc:2781] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:37.002566 11733 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:37.003430 11960 raft_consensus.cc:686] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 4 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:37.026055 12037 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:37.122989 11957 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:55:37.075448 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 371219us
I20240426 09:55:37.001927 11741 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:37.005383 11805 raft_consensus.cc:2781] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:36.709525 12011 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.029721 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.029597 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:37.102416 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 99399.4us
I20240426 09:55:37.169014 10013 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:37.098663 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:36.766760 12058 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.327s
I20240426 09:55:37.106103 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.106668 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.161181 11805 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:37.181008 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.181305 11805 raft_consensus.cc:3037] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:37.182106 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.181905 11960 consensus_queue.cc:237] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:37.036265 11979 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:37.118676 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.125864 10007 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:37.126217 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.185484 10083 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:37.159874 11957 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.188529 11957 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:37.167744 11741 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:37.191260 12066 log.cc:924] Time spent T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20240426 09:55:37.195185 11955 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:37.198079 12011 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:37.198412 11741 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:37.198640 12031 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:37.161435 12037 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.094751 11771 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:37.201109 12037 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:37.109182 11858 raft_consensus.cc:2726] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:37.075820 11956 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:37.132469 10014 raft_consensus.cc:1261] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 09:55:37.118857 11644 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:36.691977 12040 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.133364 12063 raft_consensus.cc:686] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:37.133853 11230 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:37.123215 11937 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:37.122807 11980 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:37.135090 11733 raft_consensus.cc:3037] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:37.121606 11691 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.133997 11996 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:37.170518 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.239898 12063 consensus_queue.cc:237] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.223517 11230 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:37.243800 11943 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:37.219259 11858 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:37.245353 11858 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:37.247337 11858 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:37.248663 11805 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:37.217257 11644 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:37.252233 9941 raft_consensus.cc:1226] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:37.129411 10055 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
W20240426 09:55:37.253594 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 124242us
I20240426 09:55:37.198092 11970 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:37.214164 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.214541 10012 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34938 (request call id 7) took 1122 ms. Trace:
I20240426 09:55:37.289774 11970 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.290531 10012 rpcz_store.cc:276] 0426 09:55:36.091800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:36.102513 (+ 10713us) service_pool.cc:225] Handling call
0426 09:55:37.214358 (+1111845us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b540030edd4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c40d53 00007f9ca7b8b48a 00007f9ca127acd4 00007f9ca127ac62 00007f9ca127ab91 00007f9ca127ab0c 00007f9ca1279dc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:37.214482 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":255931776}
I20240426 09:55:37.218434 10016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 1424) took 3997 ms. Trace:
I20240426 09:55:37.235271 11691 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:37.292084 11970 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:37.227876 11980 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.294080 11980 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:37.291661 10016 rpcz_store.cc:276] 0426 09:55:33.217789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:33.229945 (+ 12156us) service_pool.cc:225] Handling call
0426 09:55:34.821659 (+1591714us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:55:35.399436 (+577777us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:37.215086 (+1815650us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b540030f2d0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c45b99 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:37.215214 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3660004096}
I20240426 09:55:37.228127 11937 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.238607 10085 raft_consensus.cc:2396] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240426 09:55:37.239279 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:37.199156 12031 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.296558 11937 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:37.297389 12031 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:37.282527 11741 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:37.239342 12040 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:37.250669 11805 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:37.288286 11230 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.235927 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:37.311645 11230 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:37.313700 11782 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:37.326184 11938 raft_consensus.cc:3032] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:37.326653 11938 raft_consensus.cc:729] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:37.327512 11938 consensus_queue.cc:260] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.252899 11952 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:37.240000 11996 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:37.336794 10084 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 09:55:37.304951 11741 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.341189 11741 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:37.303078 11943 raft_consensus.cc:2781] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:37.343048 11943 raft_consensus.cc:686] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:37.343926 11943 consensus_queue.cc:237] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.356573 11996 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:37.359735 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:37.301483 11733 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.367833 10008 raft_consensus.cc:2379] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 3: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:37.370527 11733 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:37.401129 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 212108us
I20240426 09:55:37.300629 10055 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:37.413373 9939 raft_consensus.cc:1226] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6. Ops: []
W20240426 09:55:37.414870 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 114315us
I20240426 09:55:37.415732 10055 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:37.418243 10055 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:37.425088 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:37.425985 9939 raft_consensus.cc:2379] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 5: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:37.438879 10055 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:37.440802 10055 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:37.417340 11877 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:37.442700 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.424708 11996 raft_consensus.cc:2781] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:37.448021 9932 raft_consensus.cc:1226] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20240426 09:55:37.449388 10008 raft_consensus.cc:1226] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:37.438077 12040 raft_consensus.cc:2726] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 09:55:37.441445 11679 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.007s sys 0.008s
I20240426 09:55:37.445094 11976 consensus_queue.cc:1035] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20240426 09:55:37.457265 10583 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:37.445583 11688 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:37.460268 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.420078 11771 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:37.459053 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.446835 11996 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:37.458931 10010 pending_rounds.cc:76] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting all ops after (but not including) 3
I20240426 09:55:37.445827 11828 raft_consensus.cc:2781] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:37.463106 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.447347 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:37.466189 11688 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:37.471665 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:37.461541 11679 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.694s user 0.007s sys 0.009s
I20240426 09:55:37.466594 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.479105 10012 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:37.468995 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.472091 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.472098 10583 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:37.467449 10010 pending_rounds.cc:97] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20240426 09:55:37.469535 11828 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:37.474217 11688 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:37.476138 11771 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:37.476534 10081 raft_consensus.cc:2379] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 6: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:37.477435 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:37.499290 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.480410 10010 raft_consensus.cc:2866] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:37.480796 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.447573 11829 consensus_queue.cc:1035] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20240426 09:55:37.481931 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.502100 9940 raft_consensus.cc:2379] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 6: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:37.484704 10009 raft_consensus.cc:2396] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20240426 09:55:37.484958 11828 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:37.464073 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:37.466333 12040 consensus_queue.cc:1046] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20240426 09:55:37.469166 11996 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:37.508095 11967 raft_consensus.cc:3032] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:37.479993 10014 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:37.510349 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.511029 10090 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:37.515497 11644 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.483003 10055 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:37.502816 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.485226 12046 raft_consensus.cc:2726] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:37.504534 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.508493 11967 raft_consensus.cc:729] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:37.519783 9933 raft_consensus.cc:2396] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20240426 09:55:37.520363 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:37.520948 9940 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:37.519273 11691 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:37.525293 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1428) took 1198 ms. Trace:
I20240426 09:55:37.525676 9937 rpcz_store.cc:276] 0426 09:55:36.327238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:36.336145 (+ 8907us) service_pool.cc:225] Handling call
0426 09:55:37.525127 (+1188982us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54003769d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:37.525239 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":261560576}
W20240426 09:55:37.535929 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 53024.2us
I20240426 09:55:37.545060 10055 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:37.546023 10055 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:37.525692 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.519263 11644 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:37.551918 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.553929 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:37.554348 9938 raft_consensus.cc:2362] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4. Re-sending same reply.
I20240426 09:55:37.543850 11829 raft_consensus.cc:2726] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:37.557096 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.570323 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.534499 12068 raft_consensus.cc:2726] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:37.550029 11943 raft_consensus.cc:2781] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:37.573725 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.575105 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.550511 12039 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:55:37.577111 9934 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:37.581154 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.567376 11972 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:37.536496 11691 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.598846 12040 raft_consensus.cc:3032] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:37.599287 12040 raft_consensus.cc:729] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
W20240426 09:55:37.599555 12079 log.cc:924] Time spent T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.131s user 0.002s sys 0.000s
I20240426 09:55:37.599857 11691 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:37.600075 12040 consensus_queue.cc:260] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.577420 11943 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:37.601239 12040 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:37.601186 11943 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:37.602787 11972 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.603722 11947 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:37.604642 11972 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:37.579090 12039 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 09:55:37.595244 12082 log.cc:924] Time spent T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20240426 09:55:37.604699 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 1460) took 1123 ms. Trace:
I20240426 09:55:37.607401 10009 rpcz_store.cc:276] 0426 09:55:36.481710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:36.481865 (+ 155us) service_pool.cc:225] Handling call
0426 09:55:37.604540 (+1122675us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54003bd8d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:37.604649 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2303792000}
I20240426 09:55:37.598039 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.608850 9938 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:37.529829 11996 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.587942 9940 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:37.615445 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:37.592343 11688 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.617689 9940 raft_consensus.cc:2379] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 5: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:37.626096 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.627266 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.628312 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:37.628741 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.629304 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.629815 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:37.630033 9938 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
W20240426 09:55:37.632622 12085 log.cc:924] Time spent T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.158s user 0.000s sys 0.000s
I20240426 09:55:37.616189 11628 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:37.629370 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:37.636289 9934 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:37.637146 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.629767 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.645140 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.611568 12039 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:37.656473 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.654667 10086 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:37.658001 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.658708 10087 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:37.664137 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.664665 10087 raft_consensus.cc:2454] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:55:37.667011 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.668347 9934 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:37.617000 11956 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:37.671279 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.669112 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.672382 10085 raft_consensus.cc:2379] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 6: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:37.572141 11967 consensus_queue.cc:260] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.675813 10056 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:37.677201 10082 raft_consensus.cc:2379] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 5: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:37.635306 11628 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:37.683457 10056 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:37.685354 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.685626 10013 raft_consensus.cc:1226] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:37.686419 10012 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:37.691509 10014 raft_consensus.cc:1261] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 09:55:37.693001 11417 consensus_queue.cc:1035] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:37.696051 10055 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:37.685894 11967 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:37.704490 10086 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:37.707067 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.707170 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.708827 10085 raft_consensus.cc:2379] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 7: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:37.708081 10088 raft_consensus.cc:2454] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 1.
I20240426 09:55:37.708863 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.678685 11976 consensus_queue.cc:1046] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20240426 09:55:37.676796 11956 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:37.720417 10089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 1221) took 1987 ms (client timeout 1855 ms). Trace:
I20240426 09:55:37.722961 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:37.718820 10082 raft_consensus.cc:1226] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:37.724632 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.725244 10083 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
W20240426 09:55:37.720786 10089 rpcz_store.cc:269] 0426 09:55:35.732861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:35.733096 (+ 235us) service_pool.cc:225] Handling call
0426 09:55:37.720250 (+1987154us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b5400252454. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:37.720359 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4083409664}
I20240426 09:55:37.716506 11691 raft_consensus.cc:2726] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:37.723117 11741 consensus_queue.cc:1046] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20240426 09:55:37.729290 11741 raft_consensus.cc:3032] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:37.729616 11741 raft_consensus.cc:729] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
W20240426 09:55:37.730532 12092 log.cc:924] Time spent T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20240426 09:55:37.739346 11996 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:37.740063 9982 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:37.723613 12094 raft_consensus.cc:2726] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:37.742287 11733 raft_consensus.cc:2726] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:37.741472 9982 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:37.714303 11877 raft_consensus.cc:2726] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:37.754407 10083 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:55:37.761859 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:37.771250 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.766502 11230 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:37.777583 11230 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.778988 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.779968 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.780550 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.778512 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:37.783905 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:37.813829 11230 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:37.814565 10007 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:37.816552 9982 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:37.761586 11688 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:37.756484 9904 connection.cc:232] RPC call timeout handler was delayed by -1.2568804750000027s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
W20240426 09:55:37.771147 12075 log.cc:924] Time spent T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.305s user 0.000s sys 0.002s
I20240426 09:55:37.820889 11230 raft_consensus.cc:2781] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:37.825976 11230 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:37.826297 11230 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
W20240426 09:55:37.761219 12076 log.cc:924] Time spent T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.315s user 0.000s sys 0.003s
I20240426 09:55:37.822611 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.818679 9932 raft_consensus.cc:1226] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:37.827018 9934 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:37.757153 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:37.827181 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.827824 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.823621 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.829624 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.823992 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.823266 9982 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:37.830837 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.748448 11948 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:37.730372 11741 consensus_queue.cc:260] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.832921 9982 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:37.835836 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.836649 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.837154 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:37.837678 9940 raft_consensus.cc:2362] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5. Re-sending same reply.
I20240426 09:55:37.766712 11980 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:37.844481 9938 raft_consensus.cc:1226] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:37.844588 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.840613 9940 raft_consensus.cc:1226] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:37.841683 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.844671 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.845930 9934 raft_consensus.cc:2379] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 6: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:37.846705 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.847712 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.842275 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.840675 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.838083 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:37.842250 9933 raft_consensus.cc:1261] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:37.865401 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.764328 11828 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.869453 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 1424) took 1543 ms. Trace:
W20240426 09:55:37.775563 12086 log.cc:924] Time spent T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.302s user 0.000s sys 0.003s
I20240426 09:55:37.768601 11951 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:37.761734 11937 raft_consensus.cc:2781] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:37.870215 9939 raft_consensus.cc:1226] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 6. Current term is 7. Ops: []
I20240426 09:55:37.869871 9935 rpcz_store.cc:276] 0426 09:55:36.326183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:36.337088 (+ 10905us) service_pool.cc:225] Handling call
0426 09:55:37.869301 (+1532213us) spinlock_profiling.cc:243] Waited 353 ms on lock 0x7b54001aed54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:55:37.869399 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":812986752}
I20240426 09:55:37.871373 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:37.865442 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 169443us
I20240426 09:55:37.932801 9933 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:37.820307 11943 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.935146 10620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51158 (request call id 200) took 1579 ms. Trace:
I20240426 09:55:37.756075 12067 consensus_queue.cc:1046] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.115s
I20240426 09:55:37.898401 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:37.932044 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 59974.2us
I20240426 09:55:37.937980 10082 raft_consensus.cc:2354] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5.
I20240426 09:55:37.900018 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.915851 11230 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.942276 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.943164 11230 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:37.834285 11721 raft_consensus.cc:2726] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:37.872442 11968 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:37.937170 10055 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:37.935513 10620 rpcz_store.cc:276] 0426 09:55:36.355900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:36.356032 (+ 132us) service_pool.cc:225] Handling call
0426 09:55:36.967209 (+611177us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b5800047a98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:55:36.998503 (+ 31294us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:55:37.003585 (+ 5082us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:55:37.003625 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:55:37.004047 (+ 422us) log.cc:842] Serialized 26 byte log entry
0426 09:55:37.220671 (+216624us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:55:37.220707 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:55:37.220765 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:55:37.737985 (+517220us) spinlock_profiling.cc:243] Waited 40.1 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:55:37.934980 (+196995us) raft_consensus.cc:1671] finished
0426 09:55:37.935000 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:55:37.935100 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":690,"raft.queue_time_us":11442,"raft.run_cpu_time_us":44,"raft.run_wall_time_us":44,"spinlock_wait_cycles":116292992,"thread_start_us":179190,"threads_started":1,"wal-append.queue_time_us":383554}
I20240426 09:55:37.949039 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:37.931281 11951 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:37.955513 11951 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.956372 12094 consensus_queue.cc:1046] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:37.957171 12094 raft_consensus.cc:3032] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:37.893447 12031 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:37.957499 12094 raft_consensus.cc:729] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:37.960162 12031 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.950381 10084 raft_consensus.cc:1226] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:37.961978 12031 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:37.952023 12074 log.cc:924] Time spent T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.472s user 0.002s sys 0.000s
W20240426 09:55:37.952714 12095 log.cc:924] Time spent T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20240426 09:55:37.842991 11948 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.955039 11628 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:37.955883 11828 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:37.967607 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.968443 11814 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:37.968638 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.969615 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.873531 11963 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:55:37.958273 12094 consensus_queue.cc:260] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.970696 12094 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:37.963459 11968 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.974681 11739 consensus_queue.cc:1046] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:37.974984 11963 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:37.975387 11963 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:37.975654 11733 raft_consensus.cc:3032] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:37.975991 11733 raft_consensus.cc:729] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:37.976089 10086 raft_consensus.cc:1226] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20240426 09:55:37.976742 11733 consensus_queue.cc:260] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.978036 11733 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:37.979058 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:37.979475 11948 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:37.962836 11771 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:37.984635 11771 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:37.985011 11771 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
W20240426 09:55:37.985641 12087 log.cc:924] Time spent T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.508s user 0.001s sys 0.001s
I20240426 09:55:37.987962 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.988996 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:37.989611 10008 raft_consensus.cc:2379] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 6: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:37.990211 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:37.992383 11943 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:37.953419 11830 raft_consensus.cc:3032] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:37.957521 11951 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:37.867383 11980 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.865955 12055 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:37.994755 11830 raft_consensus.cc:729] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:37.996943 12055 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.997587 11938 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:37.998117 11830 consensus_queue.cc:260] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:37.953568 11956 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:38.003454 11830 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:38.005285 11978 raft_consensus.cc:3032] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:38.005671 11978 raft_consensus.cc:729] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:38.006431 11978 consensus_queue.cc:260] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:38.007767 11978 raft_consensus.cc:3037] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:38.008181 10013 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:38.010917 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:38.000031 11980 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:38.013005 12055 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:38.015574 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 51293.9us
I20240426 09:55:38.016860 9939 raft_consensus.cc:1261] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:55:38.018205 10008 raft_consensus.cc:1226] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: []
W20240426 09:55:38.020661 9838 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50240 (request call id 200) took 15140 ms (client timeout 15000 ms). Trace:
I20240426 09:55:38.023954 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:37.877346 11741 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:37.982446 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:37.984905 12091 log.cc:924] Time spent T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.308s user 0.002s sys 0.000s
W20240426 09:55:38.033496 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 333915us
I20240426 09:55:37.970558 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:38.037422 9979 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.040256 10010 raft_consensus.cc:2354] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5.
I20240426 09:55:38.042321 10011 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:55:38.048008 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:38.050700 9904 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.052090 9904 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:38.053248 9904 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.055197 10056 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.056290 10056 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.049063 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:38.057386 10056 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.054489 9979 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
W20240426 09:55:38.056421 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 105643us
I20240426 09:55:38.062004 10010 raft_consensus.cc:1226] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:37.944906 11937 raft_consensus.cc:686] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
W20240426 09:55:38.021023 9838 rpcz_store.cc:269] 0426 09:55:22.880166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:22.880603 (+ 437us) service_pool.cc:225] Handling call
0426 09:55:38.020600 (+15139997us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:31.436786 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:31.676272 (+239486us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:31.676320 (+ 48us) write_op.cc:427] Acquired schema lock
0426 09:55:31.676336 (+ 16us) tablet.cc:660] Decoding operations
0426 09:55:31.677130 (+ 794us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:31.677235 (+ 105us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:31.677260 (+ 25us) tablet.cc:683] Acquiring locks for 7 operations
0426 09:55:31.677885 (+ 625us) tablet.cc:699] Row locks acquired
0426 09:55:31.677924 (+ 39us) write_op.cc:260] PREPARE: finished
0426 09:55:31.678074 (+ 150us) write_op.cc:270] Start()
0426 09:55:31.678204 (+ 130us) write_op.cc:276] Timestamp: P: 1714125331678021 usec, L: 0
0426 09:55:31.678224 (+ 20us) op_driver.cc:331] REPLICATION: starting
0426 09:55:31.679541 (+ 1317us) log.cc:842] Serialized 2864 byte log entry
0426 09:55:32.571973 (+892432us) op_driver.cc:444] REPLICATION: finished
0426 09:55:33.198439 (+626466us) write_op.cc:301] APPLY: starting
0426 09:55:33.206079 (+ 7640us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:33.241333 (+ 35254us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:33.260583 (+ 19250us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:33.466056 (+205473us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:33.508748 (+ 42692us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0426 09:55:33.520873 (+ 12125us) write_op.cc:312] APPLY: finished
0426 09:55:33.557016 (+ 36143us) log.cc:842] Serialized 58 byte log entry
0426 09:55:33.652277 (+ 95261us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:33.662881 (+ 10604us) write_op.cc:446] Released schema lock
0426 09:55:33.680354 (+ 17473us) write_op.cc:341] FINISH: Updating metrics
0426 09:55:33.680609 (+ 255us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:33.680637 (+ 28us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":1657,"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":619015,"apply.run_cpu_time_us":13709,"apply.run_wall_time_us":487259,"num_ops":7,"prepare.queue_time_us":135864,"prepare.run_cpu_time_us":13204,"prepare.run_wall_time_us":452837,"raft.queue_time_us":566605,"raft.run_cpu_time_us":17864,"raft.run_wall_time_us":488802,"replication_time_us":893660,"thread_start_us":896899,"threads_started":5,"wal-append.queue_time_us":323289,"wal-append.run_cpu_time_us":14588,"wal-append.run_wall_time_us":2428952}]]}
I20240426 09:55:38.063645 9979 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.064246 10082 raft_consensus.cc:1226] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:38.072409 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:38.069046 11956 raft_consensus.cc:2726] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:55:38.073449 11967 raft_consensus.cc:2726] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.076627 10585 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.069444 11829 raft_consensus.cc:2726] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:38.054049 11968 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:38.077999 11971 raft_consensus.cc:2726] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:38.076010 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:38.056648 11814 raft_consensus.cc:2781] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:38.082659 11814 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:38.083052 11814 raft_consensus.cc:3037] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:38.055516 12055 raft_consensus.cc:2726] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:38.090517 10056 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.091807 9979 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.064725 12081 raft_consensus.cc:2726] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.096302 10056 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:38.098501 9979 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.111457 9979 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.115917 9979 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.099931 11769 raft_consensus.cc:2726] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:38.117269 11948 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:38.104287 9839 catalog_manager.cc:5484] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 3 to 4, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 4 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:38.119385 9839 catalog_manager.cc:5484] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 3 to 5, leader changed from 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 5 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:38.096642 12098 raft_consensus.cc:2726] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:38.162137 9904 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:38.162305 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 66075.5us
I20240426 09:55:38.166646 9904 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:38.168696 10056 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:38.126116 9979 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.105196 11688 raft_consensus.cc:2726] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.111627 11733 raft_consensus.cc:2726] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.169623 11230 raft_consensus.cc:2726] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240426 09:55:38.170188 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 44162us
W20240426 09:55:38.167670 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 67570.4us
I20240426 09:55:38.171898 9979 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.174499 9904 leader_election.cc:398] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5.
I20240426 09:55:38.174849 9904 leader_election.cc:401] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20240426 09:55:38.176273 10056 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.177554 11230 raft_consensus.cc:2726] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:38.178856 9979 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:38.179836 9979 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.159111 11721 raft_consensus.cc:2726] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:38.182430 10056 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.160857 11739 raft_consensus.cc:2726] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:55:38.177860 9839 catalog_manager.cc:5484] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 1 to 5, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 5 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:38.192279 9839 catalog_manager.cc:5484] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 1 to 3, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 3 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:38.188071 11733 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:38.110895 11980 consensus_queue.cc:1046] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
I20240426 09:55:38.194249 11230 raft_consensus.cc:3032] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:55:38.193766 9839 catalog_manager.cc:5484] T 5c0fe10a2b3746f9909d3596580b6854 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:38.194576 11230 raft_consensus.cc:729] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:38.195313 11230 consensus_queue.cc:260] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.196300 11230 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:38.193675 11733 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:38.197341 11733 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:38.203083 9839 catalog_manager.cc:5484] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: leader changed from <none> to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 3 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:38.204193 11733 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.205055 11230 raft_consensus.cc:2726] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.205618 9979 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.110757 12032 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.074s
W20240426 09:55:38.213574 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 380705us
I20240426 09:55:38.214061 11733 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:38.214210 9982 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.214461 11230 raft_consensus.cc:2726] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:55:38.174705 11976 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:38.174989 12069 raft_consensus.cc:2726] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:38.178022 11804 raft_consensus.cc:2781] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:38.183883 11858 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:38.220448 9982 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
W20240426 09:55:38.187785 12039 consensus_meta.cc:287] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.518s user 0.007s sys 0.000s
I20240426 09:55:38.217757 9839 catalog_manager.cc:5484] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:38.220161 11230 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:38.185124 12055 consensus_queue.cc:1046] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20240426 09:55:38.186972 12040 raft_consensus.cc:2726] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240426 09:55:38.236675 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 54935.7us
W20240426 09:55:38.238634 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 56271.4us
I20240426 09:55:38.239444 9979 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.170495 11965 consensus_queue.cc:1046] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.059s
I20240426 09:55:38.241751 11721 raft_consensus.cc:2726] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.257546 9979 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.242424 11996 raft_consensus.cc:3037] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:38.236270 11980 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20240426 09:55:38.249966 11230 raft_consensus.cc:686] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:38.267490 11230 consensus_queue.cc:237] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.187911 11688 raft_consensus.cc:2781] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:38.222440 11976 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:38.275082 10011 raft_consensus.cc:1261] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:38.276019 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:38.276542 10011 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:38.278244 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:38.265520 11967 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:38.272467 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:38.279479 10007 raft_consensus.cc:2379] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:38.255363 11804 raft_consensus.cc:686] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 6 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:38.280860 11804 consensus_queue.cc:237] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.290534 11963 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.291606 10011 raft_consensus.cc:1226] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 6. Current term is 7. Ops: []
I20240426 09:55:38.292505 11963 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:38.265884 12103 raft_consensus.cc:2726] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:55:38.251169 9839 catalog_manager.cc:5484] T 1c21d6dedd1441bd977f09cdcf41d436 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:38.255196 10056 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.172451 11937 consensus_queue.cc:237] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.258754 9979 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.257994 10086 raft_consensus.cc:1261] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 09:55:38.189563 11771 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.263319 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:38.306232 10009 raft_consensus.cc:2379] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:38.263883 10583 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.264117 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:38.307996 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:55:38.308454 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 44631us
W20240426 09:55:38.309262 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 50586.8us
I20240426 09:55:38.314788 9906 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.316263 9906 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.318985 10009 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "60bebc132e4141bdbab199326986516e"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:38.241025 12039 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.319541 10009 raft_consensus.cc:595] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.320626 10009 raft_consensus.cc:603] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 09:55:38.321431 10016 raft_consensus.cc:1226] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 5. Current term is 6. Ops: []
I20240426 09:55:38.322405 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:38.272944 12031 raft_consensus.cc:3032] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:38.294284 11976 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:38.323647 10013 raft_consensus.cc:1261] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240426 09:55:38.282457 11814 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.327543 10086 raft_consensus.cc:1226] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6. Ops: []
I20240426 09:55:38.328243 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:38.329021 10081 raft_consensus.cc:2396] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 09:55:38.273115 11688 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:38.325734 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:38.337114 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:38.326006 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:38.338266 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 83141.9us
I20240426 09:55:38.326359 9906 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:38.339345 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:38.339934 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:38.340157 9906 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:38.340620 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:38.341305 9906 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.341653 10084 raft_consensus.cc:2379] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 4: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:38.302631 11967 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:38.327623 10013 pending_rounds.cc:76] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e: Aborting all ops after (but not including) 2
I20240426 09:55:38.343185 10089 raft_consensus.cc:2354] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7.
I20240426 09:55:38.330621 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:38.346480 10085 raft_consensus.cc:2379] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 5: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:38.344686 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:38.344594 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:38.347294 10081 raft_consensus.cc:2379] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 5: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:38.322172 12103 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:38.353811 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:38.348292 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:38.350322 9906 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.356133 9906 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.355302 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:38.357244 9906 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.351619 10013 pending_rounds.cc:97] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240426 09:55:38.352360 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:38.358192 10013 raft_consensus.cc:2866] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:38.343114 10056 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.359840 10016 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:38.360410 10016 raft_consensus.cc:595] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.360883 10016 raft_consensus.cc:603] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 09:55:38.322415 12032 raft_consensus.cc:2726] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:55:38.362519 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:38.364372 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:38.360996 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:38.324156 12031 raft_consensus.cc:729] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:38.366338 10082 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:38.346020 11688 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:38.350976 11967 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:38.370352 10056 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.336692 12055 raft_consensus.cc:2726] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:55:38.360065 12081 raft_consensus.cc:2726] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:55:38.336437 11956 raft_consensus.cc:2726] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.343887 11943 raft_consensus.cc:2726] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240426 09:55:38.375736 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 155370us
I20240426 09:55:38.349144 12069 raft_consensus.cc:2726] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:38.353667 11968 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:38.378101 10056 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.330102 11965 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20240426 09:55:38.365252 11891 raft_consensus.cc:2726] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.360286 12103 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:38.365415 11978 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:38.373080 11696 raft_consensus.cc:2726] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:38.386492 10055 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:38.374804 12032 raft_consensus.cc:2726] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.384745 9982 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.399443 10056 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.400015 11230 raft_consensus.cc:2726] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:38.399643 9982 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:38.406601 11230 raft_consensus.cc:2781] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:38.407127 11230 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:38.407686 11230 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:38.408782 11948 raft_consensus.cc:2726] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:38.412302 9982 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:38.414584 10056 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.416172 11739 raft_consensus.cc:2781] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:38.416692 11739 raft_consensus.cc:686] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:38.417498 11739 consensus_queue.cc:237] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.419529 12101 raft_consensus.cc:2726] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:38.420127 12101 raft_consensus.cc:2726] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.420317 12098 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:55:38.420717 12098 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:38.421026 11943 raft_consensus.cc:2726] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 09:55:38.421670 11877 consensus_queue.cc:1046] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:38.422606 11877 raft_consensus.cc:3032] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Stepping down as leader of term 6
W20240426 09:55:38.422994 12104 log.cc:924] Time spent T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20240426 09:55:38.424031 12039 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:38.424393 9979 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.424667 11950 consensus_queue.cc:1035] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:38.421027 12098 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:38.426980 11771 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:38.384049 11951 raft_consensus.cc:2726] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:38.394482 12103 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:38.394807 12031 consensus_queue.cc:260] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.431871 12031 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:38.434021 11963 raft_consensus.cc:2726] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.422932 11877 raft_consensus.cc:729] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:38.435252 11877 consensus_queue.cc:260] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.442677 9979 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.443430 12110 raft_consensus.cc:2726] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.444432 9979 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.445066 12110 raft_consensus.cc:2726] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 09:55:38.446100 9979 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.446758 11956 raft_consensus.cc:2726] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:38.448084 11951 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:38.448556 11951 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:38.448843 10089 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:50408
W20240426 09:55:38.385728 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 206776us
I20240426 09:55:38.449867 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:38.450069 9904 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.450439 11951 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:38.451133 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:38.451251 11947 raft_consensus.cc:2726] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 09:55:38.452384 11877 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:38.454993 9979 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.452581 12031 consensus_queue.cc:1046] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20240426 09:55:38.464601 12098 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.468109 12032 raft_consensus.cc:3032] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:38.468473 12032 raft_consensus.cc:729] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:38.469497 12098 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:38.469203 12032 consensus_queue.cc:260] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:38.465026 10015 raft_consensus.cc:2396] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20240426 09:55:38.470574 10056 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:38.466840 11965 consensus_queue.cc:1046] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20240426 09:55:38.471520 10013 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:38.471776 11965 raft_consensus.cc:3032] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:38.471993 12032 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:38.463254 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:38.472565 10056 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.475031 12101 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:38.475507 12101 raft_consensus.cc:686] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:38.476276 12101 consensus_queue.cc:237] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.477902 11828 raft_consensus.cc:2726] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:55:38.480504 11771 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 09:55:38.477828 9979 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.452323 9904 leader_election.cc:398] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7.
I20240426 09:55:38.485559 9904 leader_election.cc:401] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20240426 09:55:38.456449 11230 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.385138 12055 raft_consensus.cc:2726] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:38.487344 10011 raft_consensus.cc:1226] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:38.486061 10013 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:55:38.489039 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:38.491067 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:38.492041 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:38.492280 9979 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:38.493140 10083 raft_consensus.cc:1261] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:38.494427 11230 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:38.496206 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:38.499311 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:55:38.499856 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 142693us
I20240426 09:55:38.500602 9906 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.502012 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:38.500555 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:38.503365 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:38.503659 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:38.504194 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:38.507203 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:38.508285 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:38.511180 10081 raft_consensus.cc:1226] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:38.517607 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:38.519129 9906 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.520334 9906 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:38.524435 10086 raft_consensus.cc:1226] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:38.463819 11814 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:38.528440 10055 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:38.532080 9937 raft_consensus.cc:1261] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:38.488446 11956 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:38.490644 11947 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 7
I20240426 09:55:38.482383 11965 raft_consensus.cc:729] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:38.394438 11938 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.122s
I20240426 09:55:38.538800 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:38.502489 10013 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:38.544335 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:38.544636 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:38.524914 11891 raft_consensus.cc:2726] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:38.547070 11691 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:38.559566 10055 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:38.560364 11995 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:38.569478 9933 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:38.599260 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:55:38.571498 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 50056.6us
I20240426 09:55:38.526329 11948 raft_consensus.cc:2726] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.576233 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:38.601361 9934 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:38.577036 10056 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.577977 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:38.494189 11967 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.603746 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:38.535878 11741 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:38.606184 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:38.607300 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:38.610817 9906 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:38.592113 10013 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
W20240426 09:55:38.579049 12095 log.cc:924] Time spent T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20240426 09:55:38.616050 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:38.530400 12081 raft_consensus.cc:2726] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.619800 10055 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
W20240426 09:55:38.629419 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 52479.7us
I20240426 09:55:38.630815 10056 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.632520 10056 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.626225 9906 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.624718 9934 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:38.623538 12069 consensus_queue.cc:1046] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20240426 09:55:38.641476 11996 raft_consensus.cc:2726] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:55:38.642386 10084 raft_consensus.cc:1226] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 2. Current term is 3. Ops: []
I20240426 09:55:38.608709 12048 raft_consensus.cc:3032] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:38.606657 11741 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:38.647202 10056 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.651162 10056 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:38.623687 11995 raft_consensus.cc:686] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:38.636404 12037 raft_consensus.cc:2726] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240426 09:55:38.656191 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 113466us
I20240426 09:55:38.605964 11828 consensus_queue.cc:1046] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20240426 09:55:38.643947 11774 raft_consensus.cc:2726] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.624128 11891 consensus_queue.cc:1046] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20240426 09:55:38.605010 12032 consensus_queue.cc:1035] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20240426 09:55:38.661381 9982 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.643489 12039 raft_consensus.cc:2726] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.648643 11679 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:38.672492 9982 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.644127 12118 raft_consensus.cc:2726] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:38.645941 12048 raft_consensus.cc:729] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:38.653934 11830 raft_consensus.cc:2726] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.653107 11741 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:38.678105 9982 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.664425 11858 raft_consensus.cc:2726] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5.
I20240426 09:55:38.670810 11996 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:38.643150 11976 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.664613 12063 raft_consensus.cc:2726] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:55:38.662127 11829 raft_consensus.cc:2726] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.654619 11943 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20240426 09:55:38.676198 11980 raft_consensus.cc:2726] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.703174 11679 raft_consensus.cc:686] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:38.703994 11828 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:38.712046 11995 consensus_queue.cc:237] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.713466 11967 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:38.637954 11965 consensus_queue.cc:260] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.720396 9906 leader_election.cc:398] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5.
I20240426 09:55:38.721638 12048 consensus_queue.cc:260] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.730046 12103 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.737023 10089 raft_consensus.cc:595] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
W20240426 09:55:38.785077 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 64831.5us
I20240426 09:55:38.741129 11688 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.681452 11938 raft_consensus.cc:3032] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:38.794849 11688 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:38.794114 11679 consensus_queue.cc:237] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.794833 11956 raft_consensus.cc:2726] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.792928 10089 raft_consensus.cc:603] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 09:55:38.796398 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:38.755342 11804 raft_consensus.cc:2726] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.681267 11948 raft_consensus.cc:2726] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 09:55:38.782166 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 140499us
I20240426 09:55:38.795015 11938 raft_consensus.cc:729] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:38.754871 9936 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
W20240426 09:55:38.757455 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 79423.6us
W20240426 09:55:38.755549 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 171154us
I20240426 09:55:38.758121 11733 raft_consensus.cc:2726] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:38.802196 11891 consensus_queue.cc:1046] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 09:55:38.766481 11996 raft_consensus.cc:686] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:38.802260 9906 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.809489 9941 raft_consensus.cc:1226] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240426 09:55:38.815404 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:38.816270 9982 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.814288 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:38.821245 11976 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:38.820969 11996 consensus_queue.cc:237] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.798128 11968 consensus_queue.cc:1046] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20240426 09:55:38.797211 11828 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:38.825126 11828 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:38.825374 12103 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:38.828145 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:38.829955 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:38.799252 11978 raft_consensus.cc:2726] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:38.811517 12069 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:38.811005 11877 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:38.844187 9906 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.847653 9906 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.843235 9982 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.849653 9982 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.841257 11938 consensus_queue.cc:260] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.841112 12069 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.857606 9982 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.860538 12143 raft_consensus.cc:2726] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:38.845793 12131 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:38.864204 12131 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:38.864498 11938 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:38.843047 11877 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.865665 10056 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.855954 12103 raft_consensus.cc:2726] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.867918 11951 raft_consensus.cc:2726] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:38.868620 12069 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:38.854578 12118 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:38.869514 12118 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:38.869874 12118 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:38.870038 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:38.870412 11965 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:55:38.873051 12127 log.cc:924] Time spent T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20240426 09:55:38.864585 12131 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:38.843740 11721 raft_consensus.cc:2726] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 09:55:38.842530 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:38.878034 9906 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.879318 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:38.881826 9906 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:38.882408 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:38.882934 9935 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:38.866719 11877 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:38.842599 11814 consensus_queue.cc:1046] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20240426 09:55:38.876291 11741 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.947006 11858 raft_consensus.cc:2726] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240426 09:55:38.948040 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 66327.8us
I20240426 09:55:38.948297 12135 raft_consensus.cc:3032] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 09:55:38.948727 12135 raft_consensus.cc:729] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:38.949470 12135 consensus_queue.cc:260] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.950999 12055 raft_consensus.cc:3032] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:38.951364 12055 raft_consensus.cc:729] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:38.927865 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:38.895136 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:38.899230 10054 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
W20240426 09:55:38.958056 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 58920.9us
I20240426 09:55:38.924574 9937 raft_consensus.cc:1226] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:38.925364 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:38.963217 11828 raft_consensus.cc:2726] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:38.952068 12055 consensus_queue.cc:260] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:38.904662 10056 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:38.866861 11733 raft_consensus.cc:2726] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:38.887795 11947 raft_consensus.cc:2726] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7.
I20240426 09:55:38.950616 12135 raft_consensus.cc:3037] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:55:38.970754 9937 raft_consensus.cc:1261] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20240426 09:55:38.892172 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 45463us
W20240426 09:55:38.978104 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 73514.8us
I20240426 09:55:38.884917 11804 consensus_queue.cc:1035] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20240426 09:55:38.979040 10056 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.879758 11937 raft_consensus.cc:2781] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:38.884806 12142 raft_consensus.cc:2726] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:38.989904 9939 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:55:38.972944 11951 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:38.993669 9937 raft_consensus.cc:1261] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 09:55:38.982949 12055 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:39.031325 9936 raft_consensus.cc:1261] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:39.028170 10056 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:38.998045 12110 consensus_queue.cc:1046] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20240426 09:55:39.045857 11937 raft_consensus.cc:686] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:39.046471 12110 raft_consensus.cc:3032] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:39.046865 12110 raft_consensus.cc:729] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:39.046710 11937 consensus_queue.cc:237] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.047688 12110 consensus_queue.cc:260] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.048796 12110 raft_consensus.cc:3037] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:39.033530 12141 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:39.049837 12141 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:39.051836 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 73140.3us
I20240426 09:55:39.052491 9979 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:38.998303 11951 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:38.993239 11877 raft_consensus.cc:2726] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:55:39.059898 10013 raft_consensus.cc:1226] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:39.060173 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:39.058787 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:39.065503 10016 raft_consensus.cc:1261] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20240426 09:55:39.065860 10016 pending_rounds.cc:76] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting all ops after (but not including) 2
I20240426 09:55:39.055779 11771 raft_consensus.cc:2781] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:39.066094 10016 pending_rounds.cc:97] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20240426 09:55:39.067576 10016 raft_consensus.cc:2866] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:39.070151 10090 raft_consensus.cc:1226] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:39.032243 11828 consensus_queue.cc:1035] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20240426 09:55:39.072335 11830 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:39.072263 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:39.072896 11830 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.074086 11741 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:39.079383 12048 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:39.079783 12118 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.082064 10082 raft_consensus.cc:1226] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:39.067359 12125 consensus_queue.cc:1035] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.016s
I20240426 09:55:39.079170 12131 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.056239 11948 raft_consensus.cc:2726] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:55:39.087980 12131 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:39.088393 12118 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:39.063784 11951 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:39.088614 11771 raft_consensus.cc:686] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:39.092249 11771 consensus_queue.cc:237] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.033788 12130 consensus_queue.cc:1035] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.055s
I20240426 09:55:39.088898 10015 raft_consensus.cc:1226] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240426 09:55:39.103690 12141 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:39.106850 11830 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:39.129329 12144 log.cc:924] Time spent T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20240426 09:55:39.122444 11951 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.144523 10089 raft_consensus.cc:1226] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6. Ops: []
I20240426 09:55:39.151345 11951 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:39.181466 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240426 09:55:39.195829 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:39.197064 10011 raft_consensus.cc:1261] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20240426 09:55:39.197295 10015 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:39.202577 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:39.203774 10089 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:39.205945 10082 raft_consensus.cc:1261] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20240426 09:55:39.206357 10082 pending_rounds.cc:76] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee: Aborting all ops after (but not including) 2
I20240426 09:55:39.208310 10090 raft_consensus.cc:1261] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:39.206625 10082 pending_rounds.cc:97] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20240426 09:55:39.209956 10082 raft_consensus.cc:2866] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:39.197762 10011 pending_rounds.cc:76] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting all ops after (but not including) 3
I20240426 09:55:39.214388 10011 pending_rounds.cc:97] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20240426 09:55:39.214576 10011 raft_consensus.cc:2866] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:39.227176 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:39.228111 9936 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:39.229450 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:39.230204 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:39.231231 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:39.232054 10081 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:55:39.235939 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:39.241366 10055 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:39.243556 11996 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:55:39.244230 11996 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:39.244652 11996 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:39.246217 9906 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:39.263188 11804 raft_consensus.cc:2726] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:39.263475 10082 raft_consensus.cc:1261] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240426 09:55:39.262001 11976 consensus_queue.cc:1035] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20240426 09:55:39.268072 12141 consensus_queue.cc:1046] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 09:55:39.269837 12141 raft_consensus.cc:3032] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:39.270310 12141 raft_consensus.cc:729] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:39.271078 9937 raft_consensus.cc:1261] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:39.271530 12141 consensus_queue.cc:260] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.274010 12141 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:39.277211 12048 consensus_queue.cc:1035] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 09:55:39.277626 11858 consensus_queue.cc:1046] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 09:55:39.278697 11858 raft_consensus.cc:3032] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:39.279181 11858 raft_consensus.cc:729] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:39.295852 10015 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 09:55:39.287214 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240426 09:55:39.298141 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 49224.6us
I20240426 09:55:39.300185 9979 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:39.303362 10089 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:39.304916 12130 raft_consensus.cc:2726] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:55:39.304720 11980 consensus_queue.cc:1046] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 09:55:39.305932 11980 raft_consensus.cc:3032] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:39.306421 11980 raft_consensus.cc:729] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:39.307597 11980 consensus_queue.cc:260] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.308830 11980 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:39.313565 11733 consensus_queue.cc:1046] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 09:55:39.314337 11733 consensus_queue.cc:1225] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer d3a480e831d649139ff016fb68aee3ee log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240426 09:55:39.316057 11948 consensus_queue.cc:1035] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 09:55:39.316447 11230 consensus_queue.cc:1035] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:39.315722 12130 raft_consensus.cc:3032] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:39.317786 12130 raft_consensus.cc:729] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:39.318759 12130 consensus_queue.cc:260] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.320298 12130 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:39.320734 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:39.321344 10016 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:55:39.305857 12110 consensus_queue.cc:1046] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20240426 09:55:39.328718 12110 raft_consensus.cc:3032] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:39.328946 10008 raft_consensus.cc:1226] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: []
I20240426 09:55:39.332800 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:39.333715 10015 raft_consensus.cc:2354] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7.
I20240426 09:55:39.334280 9906 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:39.335309 12055 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:55:39.335734 12055 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:39.336077 12055 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:39.334586 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:39.344198 9936 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:39.349536 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:39.343369 12110 raft_consensus.cc:729] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:39.356252 12110 consensus_queue.cc:260] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.358160 12110 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:39.356365 9836 catalog_manager.cc:5484] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:39.367445 9836 catalog_manager.cc:5484] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:39.371297 9836 catalog_manager.cc:5484] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 2 to 6, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 6 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:39.377924 11858 consensus_queue.cc:260] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.373948 9836 catalog_manager.cc:5484] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 1 to 4, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 4 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:39.404079 11858 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:39.404623 9836 catalog_manager.cc:5484] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 3 to 5, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 5 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:39.407912 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:39.409503 9937 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:39.409690 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:39.406596 9836 catalog_manager.cc:5484] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 1 to 2, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 2 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:39.392769 9906 leader_election.cc:398] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7.
I20240426 09:55:39.395851 12135 consensus_queue.cc:1046] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 09:55:39.417057 9906 leader_election.cc:401] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20240426 09:55:39.410740 9941 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 09:55:39.421336 9979 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:39.424835 9937 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20240426 09:55:39.436573 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 44129.9us
W20240426 09:55:39.439589 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 96558.5us
I20240426 09:55:39.441632 10011 raft_consensus.cc:2354] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 5. Current term is 6.
I20240426 09:55:39.343710 11996 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.452268 11230 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:39.460851 11230 raft_consensus.cc:686] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:39.440656 9836 catalog_manager.cc:5484] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 4 to 5, leader changed from 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 5 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:39.438853 12048 raft_consensus.cc:2726] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7.
I20240426 09:55:39.473482 10014 raft_consensus.cc:2454] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:39.500250 9906 leader_election.cc:398] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 5. Current term is 6.
I20240426 09:55:39.503113 9906 leader_election.cc:401] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20240426 09:55:39.502126 12055 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.504350 9836 catalog_manager.cc:5484] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 2 to 3, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 3 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:39.525477 11951 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20240426 09:55:39.526289 10008 raft_consensus.cc:1261] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:39.517413 12048 raft_consensus.cc:2726] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 5. Current term is 6.
I20240426 09:55:39.516757 12143 raft_consensus.cc:3032] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:39.519109 11230 consensus_queue.cc:237] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.532558 9839 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58806 (request call id 86) took 7668 ms. Trace:
I20240426 09:55:39.542557 9939 raft_consensus.cc:2379] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 7: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:39.546216 10011 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:39.535192 12143 raft_consensus.cc:729] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:39.553959 12143 consensus_queue.cc:260] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.555281 11774 consensus_queue.cc:1035] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20240426 09:55:39.568699 11952 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:39.557083 11948 consensus_queue.cc:1035] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20240426 09:55:39.560269 12135 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:39.575207 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:39.578271 10014 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:39.574836 9906 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:39.573519 12143 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:39.571713 9836 catalog_manager.cc:5484] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:39.567026 11996 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:55:39.590440 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 44770.2us
I20240426 09:55:39.533002 9839 rpcz_store.cc:276] 0426 09:55:31.833803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:31.834029 (+ 226us) service_pool.cc:225] Handling call
0426 09:55:39.476982 (+7642953us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:38.567750 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:38.568263 (+ 513us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:38.568301 (+ 38us) write_op.cc:427] Acquired schema lock
0426 09:55:38.568319 (+ 18us) tablet.cc:660] Decoding operations
0426 09:55:38.749955 (+181636us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:38.750019 (+ 64us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:38.750049 (+ 30us) tablet.cc:683] Acquiring locks for 8 operations
0426 09:55:38.750643 (+ 594us) tablet.cc:699] Row locks acquired
0426 09:55:38.750662 (+ 19us) write_op.cc:260] PREPARE: finished
0426 09:55:38.750811 (+ 149us) write_op.cc:270] Start()
0426 09:55:38.750909 (+ 98us) write_op.cc:276] Timestamp: P: 1714125338750748 usec, L: 0
0426 09:55:38.750928 (+ 19us) op_driver.cc:331] REPLICATION: starting
0426 09:55:38.755683 (+ 4755us) log.cc:842] Serialized 3280 byte log entry
0426 09:55:38.985970 (+230287us) op_driver.cc:444] REPLICATION: finished
0426 09:55:39.150156 (+164186us) spinlock_profiling.cc:243] Waited 5.58 ms on lock 0x7b1800049a80. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fd1be78 00007f9c9fd1f351 00007f9c9fd1f309 00007f9c9fd1f299 00007f9c9fd1f261 00007f9c9fd1e55d 00007f9ca8f8fd34 00007f9ca8f8fb69 00007f9c9fcfe5f6
0426 09:55:39.150318 (+ 162us) write_op.cc:301] APPLY: starting
0426 09:55:39.150405 (+ 87us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:39.150594 (+ 189us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:39.150617 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:39.154152 (+ 3535us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:39.154356 (+ 204us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0426 09:55:39.154399 (+ 43us) write_op.cc:312] APPLY: finished
0426 09:55:39.155412 (+ 1013us) log.cc:842] Serialized 64 byte log entry
0426 09:55:39.158608 (+ 3196us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:39.158719 (+ 111us) write_op.cc:446] Released schema lock
0426 09:55:39.158761 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0426 09:55:39.159038 (+ 277us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:39.159063 (+ 25us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":16,"child_traces":[["op",{"apply.queue_time_us":154722,"apply.run_cpu_time_us":2662,"apply.run_wall_time_us":9042,"num_ops":8,"prepare.queue_time_us":231600,"prepare.run_cpu_time_us":21057,"prepare.run_wall_time_us":276685,"raft.queue_time_us":58267,"raft.run_cpu_time_us":52751,"raft.run_wall_time_us":152824,"replication_time_us":234958,"spinlock_wait_cycles":12826880,"thread_start_us":407279,"threads_started":4,"wal-append.queue_time_us":150252}]]}
I20240426 09:55:39.606899 9932 raft_consensus.cc:1261] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 09:55:39.606370 9836 catalog_manager.cc:5484] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 1 to 4, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 4 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:39.614114 10007 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 09:55:39.603984 12141 raft_consensus.cc:2781] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:39.577828 12135 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.629968 11951 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.003s
W20240426 09:55:39.631554 12160 log.cc:924] Time spent T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
I20240426 09:55:39.631709 12135 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:39.631929 10089 raft_consensus.cc:1261] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20240426 09:55:39.632380 10089 pending_rounds.cc:76] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee: Aborting all ops after (but not including) 2
I20240426 09:55:39.632608 10089 pending_rounds.cc:97] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20240426 09:55:39.634032 10014 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:55:39.634789 9836 catalog_manager.cc:5484] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:39.632833 10089 raft_consensus.cc:2866] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:39.595302 11952 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:39.655382 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:39.656394 10014 raft_consensus.cc:2379] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 7: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:39.638000 12141 raft_consensus.cc:686] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:39.660471 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:39.661262 10055 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:39.646457 12110 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:55:39.665361 9836 catalog_manager.cc:5484] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:39.669509 12131 consensus_queue.cc:1035] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
W20240426 09:55:39.669957 12161 log.cc:924] Time spent T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20240426 09:55:39.672643 11938 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:39.673198 11938 raft_consensus.cc:686] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
W20240426 09:55:39.673789 12157 log.cc:924] Time spent T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20240426 09:55:39.674130 11938 consensus_queue.cc:237] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.676839 10011 raft_consensus.cc:1261] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 09:55:39.676415 12141 consensus_queue.cc:237] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.680188 9836 catalog_manager.cc:5484] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 1 to 3, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 3 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:39.695441 11996 consensus_queue.cc:1035] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20240426 09:55:39.695753 12159 log.cc:924] Time spent T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.148s user 0.000s sys 0.001s
I20240426 09:55:39.696861 9836 catalog_manager.cc:5484] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:39.700845 10016 raft_consensus.cc:2379] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 7: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:39.684885 12055 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:39.714473 9938 raft_consensus.cc:1261] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:39.711650 9836 catalog_manager.cc:5484] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 1 to 5, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 5 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:39.716696 9941 raft_consensus.cc:1261] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:39.724373 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:39.725193 10084 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:55:39.727174 12165 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 09:55:39.732324 9836 catalog_manager.cc:5484] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 1 to 4, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 4 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:39.734045 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
W20240426 09:55:39.739791 12163 log.cc:924] Time spent T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.153s user 0.000s sys 0.001s
I20240426 09:55:39.743681 9836 catalog_manager.cc:5484] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 2 to 4, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 4 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:39.745201 9836 catalog_manager.cc:5484] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:39.701886 12110 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:39.752611 10056 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:39.765726 9836 catalog_manager.cc:5484] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 2 to 3, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 3 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
W20240426 09:55:39.779889 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 98856.5us
I20240426 09:55:39.788414 11952 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:39.789407 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:39.790086 10008 raft_consensus.cc:2454] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:55:39.791349 10585 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:39.793447 12068 raft_consensus.cc:2781] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:39.780411 11733 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:39.797832 11230 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:39.804311 9904 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:55:39.804389 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 160477us
W20240426 09:55:39.813270 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 63513.9us
I20240426 09:55:39.817097 12135 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:55:39.817557 12135 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:55:39.817905 12135 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:39.820510 12110 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:39.829308 9836 catalog_manager.cc:5484] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:39.785017 11976 raft_consensus.cc:2726] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:55:39.830344 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:39.833494 9904 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:39.806488 12068 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:39.840092 12143 raft_consensus.cc:2726] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 09:55:39.843747 11771 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20240426 09:55:39.842126 12068 raft_consensus.cc:3037] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:39.847455 11769 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.013s
I20240426 09:55:39.848172 12055 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:39.848673 12055 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.850046 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:39.838522 9836 catalog_manager.cc:5484] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 1 to 3, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 3 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:39.873123 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:39.874178 9936 raft_consensus.cc:2454] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
I20240426 09:55:39.864256 9836 catalog_manager.cc:5484] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 2 to 5, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 5 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:39.859046 12055 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:39.883162 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:39.885035 9938 raft_consensus.cc:2379] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 4: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:39.887406 10086 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:39.849231 11733 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.902392 9904 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:39.914153 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:39.920221 12068 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:55:39.923965 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 43535.2us
I20240426 09:55:39.908267 9836 catalog_manager.cc:5484] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 2 to 4, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 4 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:39.914922 10013 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:39.926910 10618 raft_consensus.cc:2354] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 4. Current term is 5.
I20240426 09:55:39.944782 12068 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:39.935078 12141 raft_consensus.cc:2781] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:39.962035 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:39.963136 10008 raft_consensus.cc:3037] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 5
I20240426 09:55:39.973443 12168 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:39.974066 12168 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:39.976420 9980 leader_election.cc:398] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Vote denied by peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313) with higher term. Message: Invalid argument: T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 4. Current term is 5.
I20240426 09:55:39.976876 9980 leader_election.cc:401] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20240426 09:55:39.960451 12141 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:39.973280 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:39.983372 9940 raft_consensus.cc:3037] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:39.984381 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:39.955327 11774 consensus_queue.cc:1035] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20240426 09:55:39.989092 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:39.992254 9938 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:39.994058 9982 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:39.995584 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:39.996308 10090 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:39.996512 11733 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:40.003194 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:40.004465 12040 consensus_queue.cc:1035] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 09:55:40.004006 9938 raft_consensus.cc:2379] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 7: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:39.991001 9941 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:39.976377 12168 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:39.982453 12141 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:40.023204 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:40.024189 10086 raft_consensus.cc:2379] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 5: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:40.018705 11733 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:40.026182 11733 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:40.026578 11733 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:40.042184 10008 raft_consensus.cc:2454] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 5.
I20240426 09:55:40.043529 9934 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240426 09:55:40.049130 12135 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.035801 9940 raft_consensus.cc:2454] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 5.
I20240426 09:55:40.053500 9979 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:40.062238 12170 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:55:40.062661 12170 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:40.063097 12170 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:40.064010 10586 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:40.064929 12173 raft_consensus.cc:2781] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:40.065392 12173 raft_consensus.cc:686] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 5 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:55:40.066208 12173 consensus_queue.cc:237] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:40.067327 11937 raft_consensus.cc:2726] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313) with higher term. Message: Invalid argument: T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 4. Current term is 5.
I20240426 09:55:40.084740 12110 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20240426 09:55:40.088599 10090 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240426 09:55:40.121088 12169 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.020s
I20240426 09:55:40.120556 12168 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:40.145555 12048 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:40.146138 12048 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.148387 12048 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:40.139969 12168 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.150844 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:40.151796 10009 raft_consensus.cc:2379] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 7: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:40.152624 12168 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:40.153918 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:40.154538 10090 raft_consensus.cc:2454] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:40.155365 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:40.156086 10089 raft_consensus.cc:2379] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 5: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:40.157536 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:40.158210 9938 raft_consensus.cc:2454] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 3.
I20240426 09:55:40.158046 12141 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.160641 9979 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:40.161526 12187 raft_consensus.cc:2781] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:40.161876 12187 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:40.162171 12187 raft_consensus.cc:3037] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:55:40.166070 12185 log.cc:924] Time spent T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20240426 09:55:40.168699 12180 log.cc:924] Time spent T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20240426 09:55:40.175447 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:40.176172 10012 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:40.181768 11733 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.185690 9906 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
W20240426 09:55:40.195731 12192 log.cc:924] Time spent T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
W20240426 09:55:40.198964 12171 log.cc:924] Time spent T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20240426 09:55:40.190402 12135 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:40.211182 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:40.251161 12118 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:40.251654 12118 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:40.251998 12118 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:40.291143 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:40.295358 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:40.295980 10016 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:40.293468 11804 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:40.334085 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:40.334830 10084 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:40.301597 11733 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:40.353098 10090 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:55:40.354310 9904 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:40.356597 10084 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:40.357672 9904 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:40.345256 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:40.363739 10082 raft_consensus.cc:2379] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 5: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:40.335750 12141 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:40.366506 9936 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:40.338436 11804 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:40.376602 11804 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:40.383136 12048 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:40.364732 12187 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.397675 12187 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:40.409724 12118 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.411728 12118 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:40.409984 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:55:40.412487 10612 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:55:40.414351 9905 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:40.395056 12141 raft_consensus.cc:2781] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:40.416488 12048 raft_consensus.cc:686] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:40.417419 12048 consensus_queue.cc:237] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.420449 12141 raft_consensus.cc:686] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:40.421250 12141 consensus_queue.cc:237] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.423170 12135 raft_consensus.cc:2781] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:55:40.423569 12135 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:40.423882 12135 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
W20240426 09:55:40.412469 12188 log.cc:924] Time spent T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.242s user 0.002s sys 0.000s
W20240426 09:55:40.427690 12195 log.cc:924] Time spent T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.252s user 0.000s sys 0.002s
I20240426 09:55:40.430985 10016 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
W20240426 09:55:40.447252 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 54833.4us
I20240426 09:55:40.448587 11771 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:40.449757 11771 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:40.457104 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 153825us
I20240426 09:55:40.465963 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:55:40.466909 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 82679.1us
I20240426 09:55:40.445871 12170 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.491746 12170 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:40.498451 10009 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:40.498113 12135 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:40.500686 11771 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:40.503868 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:40.509379 10084 raft_consensus.cc:3037] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:40.507931 11691 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:40.510100 11691 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.512147 11691 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:40.516389 11976 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:40.516867 11976 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.520387 10010 raft_consensus.cc:1261] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 09:55:40.521628 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:40.522382 10016 raft_consensus.cc:2379] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 6: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:40.525357 9941 raft_consensus.cc:1261] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 09:55:40.526724 10007 raft_consensus.cc:1261] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:40.529038 11952 consensus_queue.cc:1035] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 09:55:40.518625 11976 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:40.535183 10084 raft_consensus.cc:2454] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:40.543061 12179 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:40.543570 12179 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.545501 12179 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:40.553915 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:40.556079 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:40.556718 10009 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:40.559734 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:40.561136 10084 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:40.558312 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:40.575284 10016 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:40.577885 9906 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:40.579476 10009 raft_consensus.cc:1261] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 09:55:40.586089 10007 raft_consensus.cc:2379] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:40.590047 12032 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:40.587666 11830 consensus_queue.cc:1035] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20240426 09:55:40.584834 9979 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
W20240426 09:55:40.606582 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 248965us
I20240426 09:55:40.614061 10055 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:40.614045 9982 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:40.617038 11733 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:40.617246 12135 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:40.603571 12032 raft_consensus.cc:686] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:40.617534 11733 raft_consensus.cc:686] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:40.618415 11733 consensus_queue.cc:237] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.620004 11996 raft_consensus.cc:2781] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:40.620400 11996 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:40.620705 11996 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:40.621120 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:40.621694 10612 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:40.626101 12219 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:40.626567 12219 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.628145 11996 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.628538 12219 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:40.626040 10055 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:40.629482 11691 raft_consensus.cc:2781] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:40.629841 11691 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:40.630157 11691 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:40.631582 12032 consensus_queue.cc:237] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.631878 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:40.632578 10007 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:40.633446 9906 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:40.634223 12135 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:40.634596 12135 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:40.634927 12135 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:40.638839 11691 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.641098 11691 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:40.644167 11771 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:40.644014 12135 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.644670 11771 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.646533 11771 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:40.647737 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:40.648285 10007 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:40.654167 12221 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:40.654655 12221 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.656641 12221 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:40.657091 10084 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:55:40.588531 11952 consensus_queue.cc:1035] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20240426 09:55:40.692241 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:40.692878 10012 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:40.694134 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:40.694900 10011 raft_consensus.cc:2379] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:40.700511 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:40.701210 10082 raft_consensus.cc:2379] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 5: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:55:40.691140 10007 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:55:40.705365 9979 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:40.704847 12055 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:40.708456 10084 raft_consensus.cc:3037] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:40.611075 12048 consensus_queue.cc:1035] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.087s
I20240426 09:55:40.708889 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:40.711268 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:40.712345 9906 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:40.662326 10612 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:55:40.723532 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:40.726384 9941 raft_consensus.cc:3037] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:40.715260 10090 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:40.723237 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:40.727083 9936 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:40.728571 10089 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:55:40.732009 9979 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:40.732553 12032 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:40.725083 12187 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:40.737071 10084 raft_consensus.cc:1261] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:40.736016 9905 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:40.742493 11721 raft_consensus.cc:2781] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:40.731770 12135 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:40.715226 12055 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.741088 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:40.751580 10011 consensus_queue.cc:237] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:40.769766 9936 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:55:40.770802 9941 raft_consensus.cc:2454] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:40.755599 9836 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50252 (request call id 190) took 2801 ms. Trace:
I20240426 09:55:40.741393 12032 raft_consensus.cc:686] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 6 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:40.743319 11996 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:40.772248 9836 rpcz_store.cc:276] 0426 09:55:37.953894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:37.959393 (+ 5499us) service_pool.cc:225] Handling call
0426 09:55:40.755350 (+2795957us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:40.251429 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:40.252003 (+ 574us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:40.252045 (+ 42us) write_op.cc:427] Acquired schema lock
0426 09:55:40.252057 (+ 12us) tablet.cc:660] Decoding operations
0426 09:55:40.294606 (+ 42549us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:40.294697 (+ 91us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:40.294723 (+ 26us) tablet.cc:683] Acquiring locks for 23 operations
0426 09:55:40.296385 (+ 1662us) tablet.cc:699] Row locks acquired
0426 09:55:40.296418 (+ 33us) write_op.cc:260] PREPARE: finished
0426 09:55:40.296559 (+ 141us) write_op.cc:270] Start()
0426 09:55:40.296689 (+ 130us) write_op.cc:276] Timestamp: P: 1714125340296508 usec, L: 0
0426 09:55:40.296725 (+ 36us) op_driver.cc:331] REPLICATION: starting
0426 09:55:40.297378 (+ 653us) log.cc:842] Serialized 9068 byte log entry
0426 09:55:40.394344 (+ 96966us) op_driver.cc:444] REPLICATION: finished
0426 09:55:40.649728 (+255384us) write_op.cc:301] APPLY: starting
0426 09:55:40.649819 (+ 91us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:40.650239 (+ 420us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:40.650370 (+ 131us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:40.652483 (+ 2113us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:40.664015 (+ 11532us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=23
0426 09:55:40.664079 (+ 64us) write_op.cc:312] APPLY: finished
0426 09:55:40.664556 (+ 477us) log.cc:842] Serialized 157 byte log entry
0426 09:55:40.664726 (+ 170us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:40.664922 (+ 196us) write_op.cc:446] Released schema lock
0426 09:55:40.664990 (+ 68us) write_op.cc:341] FINISH: Updating metrics
0426 09:55:40.665242 (+ 252us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:40.665269 (+ 27us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":254556,"apply.run_cpu_time_us":5306,"apply.run_wall_time_us":16143,"num_ops":23,"prepare.queue_time_us":194738,"prepare.run_cpu_time_us":21461,"prepare.run_wall_time_us":110888,"raft.queue_time_us":36303,"raft.run_cpu_time_us":77669,"raft.run_wall_time_us":217640,"replication_time_us":97144,"spinlock_wait_cycles":280960,"thread_start_us":451377,"threads_started":4,"wal-append.queue_time_us":39663}]]}
I20240426 09:55:40.775530 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:40.776163 10013 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:40.743870 12206 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:55:40.776907 12206 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:40.777230 12206 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:40.780201 12055 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:40.782452 9939 raft_consensus.cc:1261] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 09:55:40.742089 11858 raft_consensus.cc:2781] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:40.787669 11858 raft_consensus.cc:686] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:40.738402 12187 raft_consensus.cc:686] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:40.771855 11733 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:40.791160 12032 consensus_queue.cc:237] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.742623 9835 catalog_manager.cc:5484] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 1 to 4, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 4 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:40.794008 11858 consensus_queue.cc:237] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:40.795320 12206 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.785812 11721 raft_consensus.cc:686] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:40.814457 12187 consensus_queue.cc:237] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.815562 11721 consensus_queue.cc:237] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.799505 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:40.801847 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:40.807209 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:40.810052 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:40.819844 9939 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:40.811486 10013 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:40.814663 12206 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:40.827971 10012 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:40.808596 9835 catalog_manager.cc:5484] T e11b7521039c4feba17dc65ac6c819fb P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:40.829728 9835 catalog_manager.cc:5484] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 2 to 4, leader changed from 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 4 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:40.818989 9935 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:40.832125 9835 catalog_manager.cc:5484] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:40.832878 10055 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:40.833688 12040 raft_consensus.cc:2781] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:40.834189 12040 raft_consensus.cc:686] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:40.835160 12040 consensus_queue.cc:237] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.838390 10055 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:40.838438 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:40.839259 10084 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:40.830744 11691 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:40.843465 11691 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.848284 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:40.848713 10084 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:55:40.853104 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:40.853928 9979 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:40.865906 10009 raft_consensus.cc:1261] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:40.869611 12179 raft_consensus.cc:2781] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:40.870201 12179 raft_consensus.cc:686] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:40.871166 12179 consensus_queue.cc:237] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.882108 12187 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:40.902245 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:40.902990 10009 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:40.846019 11733 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.904793 10055 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:40.885960 12055 consensus_queue.cc:1035] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20240426 09:55:40.899628 11691 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:40.911562 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:40.912215 9936 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:40.908654 11938 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:40.915398 11938 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:40.915776 11938 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:40.917155 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:40.914911 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:40.918678 10010 raft_consensus.cc:1261] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 09:55:40.930341 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:40.945231 10084 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:40.930656 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:40.946255 10086 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:40.931876 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:40.947285 10090 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 6
I20240426 09:55:40.940261 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:40.934618 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:40.953042 10056 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:40.953748 11830 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:40.954109 11830 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:40.954448 11830 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:40.932426 11733 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:40.922680 11691 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:40.928733 12048 consensus_queue.cc:1035] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20240426 09:55:40.950214 11996 consensus_queue.cc:1035] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20240426 09:55:40.942536 11938 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:40.979748 10090 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:55:40.987277 9936 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:40.986951 11691 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:41.005664 11938 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:41.006472 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:41.009716 10084 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
W20240426 09:55:41.021522 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 118244us
W20240426 09:55:41.022011 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 54136.1us
I20240426 09:55:41.023546 9904 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:41.024569 12187 raft_consensus.cc:686] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:41.025557 12187 consensus_queue.cc:237] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:41.026204 10612 raft_consensus.cc:1261] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:55:41.003010 12210 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:55:41.026403 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 71441.3us
I20240426 09:55:41.031929 10056 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
W20240426 09:55:41.031471 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 99343.3us
W20240426 09:55:41.040731 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 83483.3us
I20240426 09:55:41.029235 12032 raft_consensus.cc:2726] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:55:41.050158 10084 raft_consensus.cc:1261] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 09:55:41.032811 12210 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:41.062392 12210 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:41.062992 9941 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 7
I20240426 09:55:41.063864 11830 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:41.066124 10015 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:41.066848 10015 raft_consensus.cc:595] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:41.051050 12219 consensus_queue.cc:1035] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20240426 09:55:41.070874 11830 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:41.072556 9941 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:55:41.063766 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:41.075220 10016 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:41.045094 12226 raft_consensus.cc:2726] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:55:41.032528 11980 consensus_queue.cc:1035] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20240426 09:55:41.080169 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:41.083586 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:41.086624 10086 raft_consensus.cc:2354] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7.
I20240426 09:55:41.088074 10056 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:55:41.093912 9904 leader_election.cc:398] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7.
I20240426 09:55:41.094245 9904 leader_election.cc:401] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20240426 09:55:41.094118 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:41.094806 10089 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:55:41.096648 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:41.102952 10011 raft_consensus.cc:2379] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 6: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:41.105176 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:41.106547 9935 raft_consensus.cc:1261] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 09:55:41.110080 9979 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:41.133183 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:41.135025 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:41.135721 9933 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:41.082048 12032 consensus_queue.cc:1035] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20240426 09:55:41.103389 12219 raft_consensus.cc:2726] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7.
I20240426 09:55:41.143993 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:41.146220 10084 raft_consensus.cc:1261] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:41.157205 12135 consensus_queue.cc:1035] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:41.157503 11691 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:41.149478 10016 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:41.150060 9936 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:41.152411 12221 consensus_queue.cc:1035] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:41.162611 10055 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:41.149394 11774 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:41.158327 12218 raft_consensus.cc:979] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:41.170727 9835 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58806 (request call id 87) took 1077 ms. Trace:
I20240426 09:55:41.167044 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:41.166105 12040 raft_consensus.cc:2726] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:55:41.183537 10056 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:41.181859 11774 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:41.189868 11938 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:41.195545 11938 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:41.195866 11938 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:41.199225 11858 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:41.200047 12215 raft_consensus.cc:1067] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Signalling peer f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:41.200196 12219 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:41.199751 11858 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:41.200709 12219 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:41.202555 12219 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:41.166281 12244 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:41.220710 10056 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:41.230024 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:41.230690 9936 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:41.173956 9835 rpcz_store.cc:276] 0426 09:55:40.088901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:40.109983 (+ 21082us) service_pool.cc:225] Handling call
0426 09:55:41.166003 (+1056020us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:41.010882 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:41.014613 (+ 3731us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:41.014656 (+ 43us) write_op.cc:427] Acquired schema lock
0426 09:55:41.014674 (+ 18us) tablet.cc:660] Decoding operations
0426 09:55:41.015284 (+ 610us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:41.015357 (+ 73us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:41.015385 (+ 28us) tablet.cc:683] Acquiring locks for 4 operations
0426 09:55:41.015759 (+ 374us) tablet.cc:699] Row locks acquired
0426 09:55:41.015774 (+ 15us) write_op.cc:260] PREPARE: finished
0426 09:55:41.015921 (+ 147us) write_op.cc:270] Start()
0426 09:55:41.016019 (+ 98us) write_op.cc:276] Timestamp: P: 1714125341015870 usec, L: 0
0426 09:55:41.016041 (+ 22us) op_driver.cc:331] REPLICATION: starting
0426 09:55:41.016705 (+ 664us) log.cc:842] Serialized 1681 byte log entry
0426 09:55:41.131078 (+114373us) op_driver.cc:444] REPLICATION: finished
0426 09:55:41.151672 (+ 20594us) write_op.cc:301] APPLY: starting
0426 09:55:41.151745 (+ 73us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:41.151876 (+ 131us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:41.151963 (+ 87us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:41.152419 (+ 456us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:41.152573 (+ 154us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0426 09:55:41.152616 (+ 43us) write_op.cc:312] APPLY: finished
0426 09:55:41.152808 (+ 192us) log.cc:842] Serialized 40 byte log entry
0426 09:55:41.152961 (+ 153us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:41.153056 (+ 95us) write_op.cc:446] Released schema lock
0426 09:55:41.153100 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0426 09:55:41.153331 (+ 231us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:41.153360 (+ 29us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":365,"reported_tablets":7,"spinlock_wait_cycles":5888,"child_traces":[["op",{"apply.queue_time_us":8295,"apply.run_cpu_time_us":1920,"apply.run_wall_time_us":1920,"num_ops":4,"prepare.queue_time_us":99241,"prepare.run_cpu_time_us":2958,"prepare.run_wall_time_us":18680,"replication_time_us":110081,"thread_start_us":65455,"threads_started":1}]]}
I20240426 09:55:41.195114 11774 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:41.264505 10056 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:41.221833 12244 raft_consensus.cc:686] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:41.231768 12182 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:41.303929 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:55:41.308748 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 136473us
I20240426 09:55:41.302335 9839 catalog_manager.cc:5484] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:41.309367 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:41.310278 9839 catalog_manager.cc:5484] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 5 to 7, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 7 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:41.311817 9839 catalog_manager.cc:5484] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:41.317466 9839 catalog_manager.cc:5484] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:41.318620 10016 raft_consensus.cc:2379] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 4: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:55:41.319141 9839 catalog_manager.cc:5484] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 3 to 5, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 5 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:41.321143 9839 catalog_manager.cc:5484] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 3 to 4, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 4 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:41.323556 9839 catalog_manager.cc:5484] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 2 to 5, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 5 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:41.301319 11769 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:55:41.303241 11733 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:55:41.317514 10010 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:41.325650 9839 catalog_manager.cc:5484] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 4 to 5, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 5 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:41.335913 10014 raft_consensus.cc:1226] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 5. Current term is 6. Ops: []
I20240426 09:55:41.336992 10010 raft_consensus.cc:595] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:41.337369 10010 raft_consensus.cc:603] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 09:55:41.332569 9839 catalog_manager.cc:5484] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 5, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 5 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:41.346706 9839 catalog_manager.cc:5484] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:41.349890 9839 catalog_manager.cc:5484] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
W20240426 09:55:41.365329 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 76064.4us
I20240426 09:55:41.355187 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:41.368274 10014 raft_consensus.cc:2354] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5.
I20240426 09:55:41.366500 9839 catalog_manager.cc:5484] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 3 to 7, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 7 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:41.330089 11769 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:41.392323 11769 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:55:41.408041 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 88452.1us
I20240426 09:55:41.418267 10084 raft_consensus.cc:1261] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:41.424958 10085 raft_consensus.cc:1261] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:55:41.416514 11858 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:41.511821 12182 raft_consensus.cc:686] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
W20240426 09:55:41.527117 12250 log.cc:924] Time spent T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
W20240426 09:55:41.528337 12246 log.cc:924] Time spent T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
W20240426 09:55:41.529067 12247 log.cc:924] Time spent T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.175s user 0.000s sys 0.001s
W20240426 09:55:41.536967 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 115771us
I20240426 09:55:41.657584 12256 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:55:41.658056 12252 log.cc:924] Time spent T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.247s user 0.001s sys 0.008s
I20240426 09:55:41.655081 12217 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:41.667371 10612 raft_consensus.cc:1261] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 09:55:41.663847 12182 consensus_queue.cc:237] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:41.666512 12217 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:41.676448 12251 log.cc:924] Time spent T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.247s user 0.010s sys 0.000s
I20240426 09:55:41.677318 12217 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:41.666889 11733 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:41.678572 11733 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:41.681095 11769 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:41.648480 11938 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:41.392691 12244 consensus_queue.cc:237] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:41.651372 12253 log.cc:924] Time spent T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.224s user 0.000s sys 0.009s
I20240426 09:55:41.703166 9932 raft_consensus.cc:1261] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 09:55:41.701251 9937 raft_consensus.cc:1261] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:55:41.721246 9935 raft_consensus.cc:1261] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:55:41.719784 9835 catalog_manager.cc:5484] T 09b98bf14ce848eca1c7ca1840a516b1 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:41.699205 9941 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 5
I20240426 09:55:41.701821 9938 raft_consensus.cc:1261] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 09:55:41.741875 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:41.719014 9940 raft_consensus.cc:1261] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:55:41.755162 9941 raft_consensus.cc:1261] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 09:55:41.719638 11769 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:41.701867 9936 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:47242
I20240426 09:55:41.757179 9936 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:41.757545 9936 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:41.762511 9934 raft_consensus.cc:2396] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240426 09:55:41.768813 9906 leader_election.cc:398] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5.
I20240426 09:55:41.769070 9906 leader_election.cc:401] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20240426 09:55:41.772930 12130 raft_consensus.cc:2932] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:41.774873 11733 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:41.768054 12276 consensus_queue.cc:1046] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20240426 09:55:41.776752 11733 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:41.776935 12048 raft_consensus.cc:3032] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:41.777356 12048 raft_consensus.cc:729] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:41.778087 12048 consensus_queue.cc:260] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:41.783601 12048 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:41.778406 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:41.784721 10085 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:41.787151 12219 raft_consensus.cc:2726] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5.
I20240426 09:55:41.658073 12256 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:41.810045 9936 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:41.812288 9936 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:41.827468 10090 raft_consensus.cc:1261] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240426 09:55:41.828866 10084 raft_consensus.cc:1261] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20240426 09:55:41.831905 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 426463us
I20240426 09:55:41.708515 12259 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.109s
I20240426 09:55:41.845055 10082 raft_consensus.cc:1261] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:55:41.858047 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:41.858567 9936 raft_consensus.cc:1226] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 6. Current term is 7. Ops: []
I20240426 09:55:41.863394 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:41.861217 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:41.867182 9904 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:41.870292 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:41.875069 10084 raft_consensus.cc:1226] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6. Ops: []
I20240426 09:55:41.876530 9839 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50254 (request call id 192) took 2090 ms. Trace:
I20240426 09:55:41.877735 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:41.878268 10081 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:41.883998 11804 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:41.872282 12040 consensus_queue.cc:1035] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20240426 09:55:41.871054 11830 consensus_queue.cc:1035] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20240426 09:55:41.896876 10081 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:55:41.876926 9839 rpcz_store.cc:276] 0426 09:55:39.786687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:39.790662 (+ 3975us) service_pool.cc:225] Handling call
0426 09:55:41.385344 (+1594682us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:41.385372 (+ 28us) write_op.cc:446] Released schema lock
0426 09:55:41.876459 (+491087us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:41.375138 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:41.375632 (+ 494us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:41.375667 (+ 35us) write_op.cc:427] Acquired schema lock
0426 09:55:41.375684 (+ 17us) tablet.cc:660] Decoding operations
0426 09:55:41.376639 (+ 955us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:41.376698 (+ 59us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:41.376725 (+ 27us) tablet.cc:683] Acquiring locks for 12 operations
0426 09:55:41.377641 (+ 916us) tablet.cc:699] Row locks acquired
0426 09:55:41.377655 (+ 14us) write_op.cc:260] PREPARE: finished
0426 09:55:41.377781 (+ 126us) write_op.cc:270] Start()
0426 09:55:41.377904 (+ 123us) write_op.cc:276] Timestamp: P: 1714125341377740 usec, L: 0
0426 09:55:41.377920 (+ 16us) op_driver.cc:331] REPLICATION: starting
0426 09:55:41.378511 (+ 591us) log.cc:842] Serialized 4809 byte log entry
0426 09:55:41.380927 (+ 2416us) op_driver.cc:444] REPLICATION: finished
0426 09:55:41.381318 (+ 391us) write_op.cc:301] APPLY: starting
0426 09:55:41.381386 (+ 68us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:41.381629 (+ 243us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:41.381712 (+ 83us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:41.382681 (+ 969us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:41.382925 (+ 244us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
0426 09:55:41.382969 (+ 44us) write_op.cc:312] APPLY: finished
0426 09:55:41.383241 (+ 272us) log.cc:842] Serialized 88 byte log entry
0426 09:55:41.383379 (+ 138us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:41.383506 (+ 127us) write_op.cc:446] Released schema lock
0426 09:55:41.383552 (+ 46us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":308,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":201,"apply.run_cpu_time_us":2486,"apply.run_wall_time_us":2485,"num_ops":12,"prepare.queue_time_us":96,"prepare.run_cpu_time_us":3630,"prepare.run_wall_time_us":10116,"replication_time_us":2913}]]}
I20240426 09:55:41.904656 9933 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:55:41.905613 10085 raft_consensus.cc:1226] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 5. Current term is 6. Ops: []
I20240426 09:55:41.905822 10056 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:41.912441 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:41.913074 9934 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:41.873253 12130 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:34362 (request call id 832) took 1106 ms. Trace:
I20240426 09:55:41.914168 9904 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:41.915894 10015 raft_consensus.cc:1261] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 09:55:41.915997 12244 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:41.918093 9838 catalog_manager.cc:5484] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 2 to 5, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 5 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:41.919960 9838 catalog_manager.cc:5484] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:41.882198 12143 consensus_queue.cc:1035] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20240426 09:55:41.912101 11733 consensus_queue.cc:1035] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.016s
I20240426 09:55:41.913615 12254 consensus_queue.cc:1035] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20240426 09:55:41.927922 12254 consensus_queue.cc:1046] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:41.928835 12254 raft_consensus.cc:3032] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:55:41.929224 12254 raft_consensus.cc:729] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:41.918943 12032 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:41.931658 12032 raft_consensus.cc:686] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:41.932448 12032 consensus_queue.cc:237] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:41.935140 11938 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:41.936479 12256 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:41.907712 12262 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:41.928974 9938 raft_consensus.cc:2932] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:41.940537 10090 consensus_queue.cc:237] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:41.942963 9838 catalog_manager.cc:5484] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 1 to 4, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 4 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:41.930049 12254 consensus_queue.cc:260] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:41.947032 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:41.948798 9934 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:41.949827 10056 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:41.947681 10015 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:41.951949 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:41.953406 9933 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:55:41.953817 9936 raft_consensus.cc:1226] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20240426 09:55:41.954591 9982 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:41.969384 10010 raft_consensus.cc:1261] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240426 09:55:41.945772 12244 raft_consensus.cc:686] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:41.971463 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:41.908205 12272 consensus_queue.cc:1035] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20240426 09:55:41.960283 12217 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:55:41.964340 12242 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:41.980743 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:41.942443 12187 consensus_queue.cc:1035] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.033s
I20240426 09:55:41.969765 12254 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:41.982431 12217 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:41.995004 9979 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:42.000109 9979 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:41.992635 12242 raft_consensus.cc:686] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:41.989324 12255 consensus_queue.cc:1035] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.010s
I20240426 09:55:42.031797 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:41.999239 12217 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:42.039992 10016 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:42.041635 10016 raft_consensus.cc:595] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Received request to transfer leadership to TS d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:41.990932 12206 consensus_queue.cc:1035] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20240426 09:55:42.005873 12273 raft_consensus.cc:2726] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 09:55:41.914934 12130 rpcz_store.cc:276] 0426 09:55:40.749603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:40.749707 (+ 104us) service_pool.cc:225] Handling call
0426 09:55:40.797534 (+ 47827us) log.cc:842] Serialized 501 byte log entry
0426 09:55:41.021177 (+223643us) spinlock_profiling.cc:243] Waited 979 us on lock 0x7b180031a9e0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fc94faa 00007f9ca9383f25 00007f9c9fd19094 00007f9c9fd18985 00007f9ca2bc04b0 00007f9ca2c262a3 00007f9ca2c4c972 00007f9ca2c44daa 00007f9ca7b89349
0426 09:55:41.855948 (+834771us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":112290,"raft.queue_time_us":125157,"raft.run_cpu_time_us":1137,"raft.run_wall_time_us":19695,"spinlock_wait_cycles":2251648,"thread_start_us":98853,"threads_started":1,"wal-append.queue_time_us":110488}
W20240426 09:55:42.048624 12279 log.cc:924] Time spent T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20240426 09:55:42.049222 12242 consensus_queue.cc:237] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:42.029271 11830 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:55:42.007709 12215 raft_consensus.cc:2726] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:55:42.051072 10613 raft_consensus.cc:1261] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20240426 09:55:42.060277 12280 log.cc:924] Time spent T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.140s user 0.000s sys 0.003s
I20240426 09:55:42.060953 12245 consensus_queue.cc:1035] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 09:55:42.062530 12273 raft_consensus.cc:979] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:41.997929 12257 consensus_queue.cc:1035] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.070s
I20240426 09:55:41.981436 10618 raft_consensus.cc:2932] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:42.005681 11733 consensus_queue.cc:1046] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20240426 09:55:42.099928 10016 raft_consensus.cc:1261] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:55:42.101459 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:42.108328 9825 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb5d051be8c743a18cd550e145b6564e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:55:42.100523 11830 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:42.109333 11733 raft_consensus.cc:3032] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:42.109697 11733 raft_consensus.cc:729] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:42.110405 11733 consensus_queue.cc:260] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:42.111529 11733 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:42.112056 11830 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:42.120570 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:42.121170 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 78760.7us
I20240426 09:55:42.129644 10015 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:55:42.130209 9936 raft_consensus.cc:1261] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 09:55:41.999141 12244 consensus_queue.cc:237] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:42.103263 12257 consensus_queue.cc:1225] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer d3a480e831d649139ff016fb68aee3ee log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20240426 09:55:42.142709 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 137176us
I20240426 09:55:42.121181 12273 raft_consensus.cc:1067] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Signalling peer d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:42.153843 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
W20240426 09:55:42.154956 12247 log.cc:924] Time spent T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.055s user 0.003s sys 0.000s
W20240426 09:55:42.159195 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 152046us
I20240426 09:55:42.146540 12262 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:42.159744 11691 consensus_queue.cc:1035] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20240426 09:55:42.177461 10011 raft_consensus.cc:1226] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 7. Current term is 8. Ops: []
I20240426 09:55:42.175134 12262 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:42.179013 11996 consensus_queue.cc:1035] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20240426 09:55:42.244343 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:42.217396 11858 consensus_queue.cc:1046] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.032s
W20240426 09:55:42.270319 12285 log.cc:924] Time spent T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.226s user 0.000s sys 0.008s
I20240426 09:55:42.275036 10087 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8574d769068f477e859574c959d5880e"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
W20240426 09:55:42.275383 12290 log.cc:924] Time spent T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20240426 09:55:42.277658 10087 raft_consensus.cc:595] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:42.274545 12262 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:42.280447 12262 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:42.281348 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:42.220665 11691 consensus_queue.cc:1035] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20240426 09:55:42.257472 12269 consensus_queue.cc:1035] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20240426 09:55:42.254662 12217 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:42.267403 10054 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:55:42.300208 10082 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:59746
I20240426 09:55:42.307765 10082 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:42.308143 10082 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:42.312495 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:55:42.316736 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 137200us
W20240426 09:55:42.316843 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 61876.2us
I20240426 09:55:42.331617 9906 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:42.332969 12262 raft_consensus.cc:2726] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:55:42.333992 10010 raft_consensus.cc:2354] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 4. Current term is 5.
I20240426 09:55:42.338224 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:42.338922 12217 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:42.339089 9938 raft_consensus.cc:2379] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 5: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:55:42.346913 12242 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:55:42.358624 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 09:55:42.360457 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:55:42.361315 10085 pending_rounds.cc:76] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee: Aborting all ops after (but not including) 3
I20240426 09:55:42.361677 10085 pending_rounds.cc:97] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20240426 09:55:42.361902 10085 raft_consensus.cc:2866] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:42.366113 10009 raft_consensus.cc:1261] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:55:42.369524 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:42.370946 10082 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:42.374696 9979 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:55:42.378865 10055 leader_election.cc:398] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 4. Current term is 5.
I20240426 09:55:42.384500 9937 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:55:42.389447 10082 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:42.406637 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:42.409557 11938 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:42.409862 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:42.410431 10009 raft_consensus.cc:3032] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:42.410712 10009 raft_consensus.cc:729] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:42.385715 12217 raft_consensus.cc:2726] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:55:42.415454 10009 consensus_queue.cc:260] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:42.427995 9933 raft_consensus.cc:1261] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20240426 09:55:42.428397 9933 pending_rounds.cc:76] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9: Aborting all ops after (but not including) 3
I20240426 09:55:42.428611 9933 pending_rounds.cc:97] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20240426 09:55:42.428828 9933 raft_consensus.cc:2866] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:42.431087 10009 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:42.460130 10009 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:42.410862 12226 consensus_queue.cc:1035] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20240426 09:55:42.453688 12210 raft_consensus.cc:3032] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Stepping down as leader of term 7
W20240426 09:55:42.471534 12312 log.cc:924] Time spent T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20240426 09:55:42.505216 10055 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:42.471395 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:42.492208 12210 raft_consensus.cc:729] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:42.506361 11769 raft_consensus.cc:2781] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:42.529662 12210 consensus_queue.cc:260] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:42.531024 12210 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:42.533392 10009 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:55:42.501214 12182 consensus_queue.cc:1035] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20240426 09:55:42.514340 12269 raft_consensus.cc:979] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:42.539253 12269 raft_consensus.cc:1067] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:42.516284 10618 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:42.552259 10086 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 09:55:42.557744 11938 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:42.559067 12263 raft_consensus.cc:2932] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:42.559998 10014 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8574d769068f477e859574c959d5880e"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:55:42.560573 10014 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:42.560918 10014 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:42.584925 12259 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:42.587005 10618 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:42.573292 10014 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:42.590607 9933 raft_consensus.cc:2932] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
W20240426 09:55:42.652060 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 82236.9us
I20240426 09:55:42.575481 11769 raft_consensus.cc:686] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:42.650244 9835 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58806 (request call id 88) took 1343 ms. Trace:
W20240426 09:55:42.659657 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 74907us
I20240426 09:55:42.665213 10014 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:42.653669 9835 rpcz_store.cc:276] 0426 09:55:41.306408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:41.314517 (+ 8109us) service_pool.cc:225] Handling call
0426 09:55:42.650170 (+1335653us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:41.869255 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:41.937909 (+ 68654us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:41.937946 (+ 37us) write_op.cc:427] Acquired schema lock
0426 09:55:41.937961 (+ 15us) tablet.cc:660] Decoding operations
0426 09:55:41.938274 (+ 313us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:41.938328 (+ 54us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:41.938356 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0426 09:55:41.938532 (+ 176us) tablet.cc:699] Row locks acquired
0426 09:55:41.938549 (+ 17us) write_op.cc:260] PREPARE: finished
0426 09:55:41.938669 (+ 120us) write_op.cc:270] Start()
0426 09:55:41.938790 (+ 121us) write_op.cc:276] Timestamp: P: 1714125341938626 usec, L: 0
0426 09:55:41.938809 (+ 19us) op_driver.cc:331] REPLICATION: starting
0426 09:55:41.939312 (+ 503us) log.cc:842] Serialized 535 byte log entry
0426 09:55:42.208869 (+269557us) op_driver.cc:444] REPLICATION: finished
0426 09:55:42.451648 (+242779us) write_op.cc:301] APPLY: starting
0426 09:55:42.454717 (+ 3069us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:42.495453 (+ 40736us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:42.505912 (+ 10459us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:42.506177 (+ 265us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:42.506353 (+ 176us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0426 09:55:42.506397 (+ 44us) write_op.cc:312] APPLY: finished
0426 09:55:42.546937 (+ 40540us) log.cc:842] Serialized 22 byte log entry
0426 09:55:42.552475 (+ 5538us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:42.565298 (+ 12823us) write_op.cc:446] Released schema lock
0426 09:55:42.570568 (+ 5270us) write_op.cc:341] FINISH: Updating metrics
0426 09:55:42.570861 (+ 293us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:42.570898 (+ 37us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":135175,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":222601,"apply.run_cpu_time_us":2345,"apply.run_wall_time_us":131048,"num_ops":1,"prepare.queue_time_us":103,"prepare.run_cpu_time_us":2382,"prepare.run_wall_time_us":93053,"replication_time_us":260549,"thread_start_us":158135,"threads_started":1}]]}
W20240426 09:55:42.665872 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 75166.1us
W20240426 09:55:42.671165 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 68735.6us
I20240426 09:55:42.690300 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:42.693508 9939 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:42.696511 10013 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 09:55:42.709178 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:42.709790 10089 raft_consensus.cc:3032] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:42.710084 10089 raft_consensus.cc:729] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:42.717680 11769 consensus_queue.cc:237] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:42.721176 12143 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:42.725796 10015 raft_consensus.cc:1261] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20240426 09:55:42.726135 10015 pending_rounds.cc:76] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting all ops after (but not including) 3
I20240426 09:55:42.726326 10015 pending_rounds.cc:97] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20240426 09:55:42.726544 10015 raft_consensus.cc:2866] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:42.728166 12242 consensus_queue.cc:1035] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:42.710726 10089 consensus_queue.cc:260] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:42.743881 10089 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
W20240426 09:55:42.744964 12315 log.cc:924] Time spent T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.052s user 0.003s sys 0.000s
I20240426 09:55:42.777086 9823 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6151686b1a24447880e49b7d7395fcc1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:55:42.780722 9939 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:42.785607 10089 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:42.739626 10613 raft_consensus.cc:2932] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:42.800740 9979 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:42.801666 9935 raft_consensus.cc:1226] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 4. Current term is 5. Ops: []
I20240426 09:55:42.795686 10011 raft_consensus.cc:1226] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 4. Current term is 5. Ops: []
W20240426 09:55:42.774246 12314 log.cc:924] Time spent T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20240426 09:55:42.805444 9933 raft_consensus.cc:1261] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:55:42.844542 12242 consensus_queue.cc:1046] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20240426 09:55:42.845732 12206 raft_consensus.cc:2781] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:42.861119 12242 raft_consensus.cc:3032] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:42.866580 12242 raft_consensus.cc:729] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:42.913861 12242 consensus_queue.cc:260] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:42.915200 12242 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:42.873032 12206 raft_consensus.cc:686] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:42.936492 12206 consensus_queue.cc:237] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:42.978304 12306 log.cc:924] Time spent T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.135s user 0.001s sys 0.001s
W20240426 09:55:43.011373 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 159422us
I20240426 09:55:43.023355 9835 catalog_manager.cc:5484] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 2 to 7, leader changed from 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 7 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:43.031986 9835 catalog_manager.cc:5484] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 3 to 4, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New cstate: current_term: 1 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:43.030544 12263 consensus_queue.cc:1035] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20240426 09:55:43.069465 9836 catalog_manager.cc:5484] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 6 to 7, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 7 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:43.097046 9836 catalog_manager.cc:5484] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 2 to 4, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 4 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:43.109586 9836 catalog_manager.cc:5484] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:43.111986 9836 catalog_manager.cc:5484] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 2 to 5, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 5 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:43.113541 9836 catalog_manager.cc:5484] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:43.117959 9836 catalog_manager.cc:5484] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 1 to 4, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 4 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:43.121575 9836 catalog_manager.cc:5484] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:43.126295 9836 catalog_manager.cc:5484] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:43.145195 9838 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50240 (request call id 201) took 3204 ms. Trace:
I20240426 09:55:43.144026 9836 catalog_manager.cc:5484] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 1 to 7. New cstate: current_term: 7 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:43.153659 9836 catalog_manager.cc:5484] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 3 to 4, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 4 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:43.148196 9838 rpcz_store.cc:276] 0426 09:55:39.941160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:39.941757 (+ 597us) service_pool.cc:225] Handling call
0426 09:55:43.145143 (+3203386us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:41.990615 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:42.105471 (+114856us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:42.120739 (+ 15268us) write_op.cc:427] Acquired schema lock
0426 09:55:42.130458 (+ 9719us) tablet.cc:660] Decoding operations
0426 09:55:42.209293 (+ 78835us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:42.210112 (+ 819us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:42.212924 (+ 2812us) tablet.cc:683] Acquiring locks for 3 operations
0426 09:55:42.254242 (+ 41318us) tablet.cc:699] Row locks acquired
0426 09:55:42.254266 (+ 24us) write_op.cc:260] PREPARE: finished
0426 09:55:42.309547 (+ 55281us) write_op.cc:270] Start()
0426 09:55:42.316971 (+ 7424us) write_op.cc:276] Timestamp: P: 1714125342308445 usec, L: 0
0426 09:55:42.343447 (+ 26476us) op_driver.cc:331] REPLICATION: starting
0426 09:55:42.400626 (+ 57179us) log.cc:842] Serialized 1316 byte log entry
0426 09:55:42.430945 (+ 30319us) op_driver.cc:444] REPLICATION: finished
0426 09:55:42.555588 (+124643us) spinlock_profiling.cc:243] Waited 47.7 ms on lock 0x7b54000041d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c42328 00007f9ca665718e 00007f9ca6656968 00007f9ca665b251 00007f9ca665b209 00007f9ca665b199 00007f9ca665b161 00007f9ca665a45d 00007f9ca8f8fd34 00007f9ca8f8fb69
0426 09:55:42.571265 (+ 15677us) write_op.cc:301] APPLY: starting
0426 09:55:42.571335 (+ 70us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:42.571468 (+ 133us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:42.571487 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:42.571935 (+ 448us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:42.646211 (+ 74276us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0426 09:55:42.660230 (+ 14019us) write_op.cc:312] APPLY: finished
0426 09:55:42.683313 (+ 23083us) log.cc:842] Serialized 34 byte log entry
0426 09:55:42.703922 (+ 20609us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:42.711675 (+ 7753us) write_op.cc:446] Released schema lock
0426 09:55:42.741189 (+ 29514us) write_op.cc:341] FINISH: Updating metrics
0426 09:55:42.781675 (+ 40486us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:42.792537 (+ 10862us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":11955,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":140124,"apply.run_cpu_time_us":2402,"apply.run_wall_time_us":237537,"mutex_wait_us":4084,"num_ops":3,"prepare.queue_time_us":1981,"prepare.run_cpu_time_us":6916,"prepare.run_wall_time_us":568646,"replication_time_us":81423,"spinlock_wait_cycles":109811968,"thread_start_us":214684,"threads_started":1}]]}
I20240426 09:55:43.243324 10015 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240426 09:55:43.247172 12332 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20240426 09:55:43.263381 10077 tablet_service.cc:1514] Processing DeleteTablet for tablet eb5d051be8c743a18cd550e145b6564e with delete_type TABLET_DATA_TOMBSTONED (TS d3a480e831d649139ff016fb68aee3ee not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59730
I20240426 09:55:43.273180 9835 catalog_manager.cc:5484] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 3 to 5, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New cstate: current_term: 3 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:43.291369 12340 tablet_replica.cc:330] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee: stopping tablet replica
I20240426 09:55:43.292790 12340 raft_consensus.cc:2229] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 09:55:43.293697 12340 raft_consensus.cc:2258] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 09:55:43.300153 10613 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240426 09:55:43.309432 12340 ts_tablet_manager.cc:1875] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:55:43.313745 12332 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 09:55:43.335728 12340 ts_tablet_manager.cc:1888] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 09:55:43.336391 12340 log.cc:1196] T eb5d051be8c743a18cd550e145b6564e P d3a480e831d649139ff016fb68aee3ee: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-2-root/wals/eb5d051be8c743a18cd550e145b6564e
I20240426 09:55:43.357045 10006 tablet_service.cc:1514] Processing DeleteTablet for tablet 6151686b1a24447880e49b7d7395fcc1 with delete_type TABLET_DATA_TOMBSTONED (TS 826f03b2fb9c4b9483623b041a1bf80e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34362
I20240426 09:55:43.355376 9836 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50252 (request call id 191) took 1498 ms. Trace:
I20240426 09:55:43.358515 9836 rpcz_store.cc:276] 0426 09:55:41.856500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:41.864775 (+ 8275us) service_pool.cc:225] Handling call
0426 09:55:43.355311 (+1490536us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:55:43.198790 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:55:43.199201 (+ 411us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:55:43.199226 (+ 25us) write_op.cc:427] Acquired schema lock
0426 09:55:43.199242 (+ 16us) tablet.cc:660] Decoding operations
0426 09:55:43.200572 (+ 1330us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:55:43.200617 (+ 45us) write_op.cc:633] Partition lock acquired for write op
0426 09:55:43.200647 (+ 30us) tablet.cc:683] Acquiring locks for 10 operations
0426 09:55:43.201542 (+ 895us) tablet.cc:699] Row locks acquired
0426 09:55:43.201563 (+ 21us) write_op.cc:260] PREPARE: finished
0426 09:55:43.201686 (+ 123us) write_op.cc:270] Start()
0426 09:55:43.201782 (+ 96us) write_op.cc:276] Timestamp: P: 1714125343201644 usec, L: 0
0426 09:55:43.201801 (+ 19us) op_driver.cc:331] REPLICATION: starting
0426 09:55:43.202317 (+ 516us) log.cc:842] Serialized 4033 byte log entry
0426 09:55:43.289990 (+ 87673us) op_driver.cc:444] REPLICATION: finished
0426 09:55:43.304518 (+ 14528us) write_op.cc:301] APPLY: starting
0426 09:55:43.304591 (+ 73us) tablet.cc:1365] starting BulkCheckPresence
0426 09:55:43.304803 (+ 212us) tablet.cc:1368] finished BulkCheckPresence
0426 09:55:43.304909 (+ 106us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:55:43.306034 (+ 1125us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:55:43.306277 (+ 243us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0426 09:55:43.306322 (+ 45us) write_op.cc:312] APPLY: finished
0426 09:55:43.306686 (+ 364us) log.cc:842] Serialized 76 byte log entry
0426 09:55:43.331597 (+ 24911us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:43.331717 (+ 120us) write_op.cc:446] Released schema lock
0426 09:55:43.331765 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0426 09:55:43.332075 (+ 310us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:55:43.332103 (+ 28us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":5419,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":1007,"apply.run_cpu_time_us":3179,"apply.run_wall_time_us":27888,"num_ops":10,"prepare.queue_time_us":35649,"prepare.run_cpu_time_us":3291,"prepare.run_wall_time_us":3777,"replication_time_us":87552}]]}
I20240426 09:55:43.361207 9839 catalog_manager.cc:5484] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 4 to 6, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 6 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:43.365321 9839 catalog_manager.cc:5484] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 1 to 6, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 6 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:43.366914 9839 catalog_manager.cc:5484] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:43.368685 9839 catalog_manager.cc:5484] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 3 to 5, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 5 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:43.379701 9836 catalog_manager.cc:5484] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 5 to 6, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 6 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:43.381035 9839 catalog_manager.cc:5484] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 4 to 6, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 6 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:43.382660 9823 catalog_manager.cc:4830] TS d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): tablet eb5d051be8c743a18cd550e145b6564e (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:55:43.386724 9839 catalog_manager.cc:5484] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 5 to 6, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 6 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:43.389199 9839 catalog_manager.cc:5484] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:43.391654 9839 catalog_manager.cc:5484] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 6 to 7, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 7 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:43.426895 9935 raft_consensus.cc:1261] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:55:43.429359 12217 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:43.464995 12343 tablet_replica.cc:330] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e: stopping tablet replica
I20240426 09:55:43.466145 12343 raft_consensus.cc:2229] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 09:55:43.467768 12343 raft_consensus.cc:2258] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 09:55:43.515153 12343 ts_tablet_manager.cc:1875] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:55:43.521900 10081 raft_consensus.cc:1261] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:55:43.523536 12273 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:43.531615 9839 catalog_manager.cc:5484] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 5 to 6, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 6 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:43.623453 10081 consensus_queue.cc:237] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:43.624805 12343 ts_tablet_manager.cc:1888] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20240426 09:55:43.625303 12343 log.cc:1196] T 6151686b1a24447880e49b7d7395fcc1 P 826f03b2fb9c4b9483623b041a1bf80e: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-1-root/wals/6151686b1a24447880e49b7d7395fcc1
I20240426 09:55:43.631129 9825 catalog_manager.cc:4830] TS 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): tablet 6151686b1a24447880e49b7d7395fcc1 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
W20240426 09:55:43.651966 10054 consensus_peers.cc:485] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 472a3b5debc94412b63080f474ae4fed. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:43.656054 10009 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:55:43.658612 12269 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20240426 09:55:43.667002 9937 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:55:43.668885 12242 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:43.701649 12032 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:43.702152 12032 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:43.704260 12032 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:43.705261 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:43.705785 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:43.706339 10009 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 8.
I20240426 09:55:43.707378 9906 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:43.708542 12032 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:55:43.707921 10015 raft_consensus.cc:2932] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:43.709014 12032 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:43.706295 9937 raft_consensus.cc:2932] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:43.709520 12032 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:55:43.716650 12032 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:43.702255 12242 raft_consensus.cc:2932] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:43.723289 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:43.723842 10009 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:55:43.731467 12032 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:43.732328 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:43.738240 10009 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 9.
I20240426 09:55:43.739364 9906 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:43.740083 12143 raft_consensus.cc:2781] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 09:55:43.740554 12143 raft_consensus.cc:686] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [term 9 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:43.741323 12143 consensus_queue.cc:237] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:43.746254 9839 catalog_manager.cc:5484] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index -1 to 5, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 7 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:43.758292 10015 consensus_queue.cc:237] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:43.778862 10081 raft_consensus.cc:1261] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20240426 09:55:43.780169 9980 consensus_peers.cc:485] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c0134eb2ced4f5c8e56412aaee1c225. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:43.780292 12273 consensus_queue.cc:1035] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 09:55:43.800833 12206 raft_consensus.cc:2932] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:43.813040 10081 raft_consensus.cc:2932] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:43.824275 9933 raft_consensus.cc:1261] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 09:55:43.835143 10015 consensus_queue.cc:237] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:43.851642 12332 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:55:43.850447 12215 consensus_queue.cc:1035] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 09:55:43.855226 12332 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:43.857741 9838 catalog_manager.cc:5484] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index -1 to 3, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 3 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:43.898629 12332 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:43.899538 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:43.899956 12032 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:43.903805 10009 raft_consensus.cc:2354] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 8. Current term is 9.
I20240426 09:55:43.905265 10055 leader_election.cc:398] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 8. Current term is 9.
I20240426 09:55:43.905563 10055 leader_election.cc:401] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20240426 09:55:43.906387 12269 raft_consensus.cc:3037] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 9
I20240426 09:55:43.907248 12032 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:43.912846 12032 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:43.924423 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:55:43.925374 10082 raft_consensus.cc:2396] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240426 09:55:43.926378 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:55:43.939553 12269 raft_consensus.cc:2726] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 8. Current term is 9.
I20240426 09:55:43.953437 9835 catalog_manager.cc:5484] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:43.962481 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:55:43.973453 9980 consensus_peers.cc:485] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67dfdb63e28a460486dfe163f03bf820. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:43.963295 10009 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:43.979758 9938 raft_consensus.cc:1261] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:55:44.003048 9933 raft_consensus.cc:2932] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:44.006223 12206 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20240426 09:55:44.024617 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:44.031765 10082 raft_consensus.cc:1261] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 09:55:44.034595 9906 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:44.045212 12354 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:55:44.045655 12354 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:44.045992 12354 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:44.045349 11721 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20240426 09:55:44.071344 12354 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:44.072417 11721 raft_consensus.cc:2932] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:44.074703 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:44.075345 10015 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:44.079337 9938 raft_consensus.cc:2932] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:44.083515 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:44.084074 10082 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:44.085319 10015 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:55:44.086314 9906 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:44.087136 12357 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:44.087594 12357 raft_consensus.cc:686] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 6 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:44.088374 12357 consensus_queue.cc:237] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:44.093746 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 49051.1us
I20240426 09:55:44.096020 10089 consensus_queue.cc:237] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:44.103178 12354 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:44.104274 10082 raft_consensus.cc:2396] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240426 09:55:44.113307 9835 catalog_manager.cc:5484] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:44.115713 9839 catalog_manager.cc:5484] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index -1 to 4, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 4 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:44.142218 10081 raft_consensus.cc:2932] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:44.153575 10009 raft_consensus.cc:1261] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 09:55:44.153985 12332 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:55:44.154490 12332 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:44.155175 12357 consensus_queue.cc:1035] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:44.156394 12332 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:44.156971 10084 raft_consensus.cc:1261] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 09:55:44.158812 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:55:44.159019 10054 consensus_peers.cc:485] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 308d6ed4b68346be85caf2871916eab9. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:44.161798 12356 consensus_queue.cc:1035] T ec8cf6b7af0543f3a118ab747864f366 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:44.163542 10014 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240426 09:55:44.164651 9933 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240426 09:55:44.164994 12269 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:44.166726 12269 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:44.179152 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:55:44.179927 10009 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:44.182499 10055 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:44.185436 12332 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:55:44.185798 12332 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:55:44.186120 12332 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
W20240426 09:55:44.251838 12364 log.cc:924] Time spent T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20240426 09:55:44.252997 12265 raft_consensus.cc:2932] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:44.243854 12365 ts_tablet_manager.cc:902] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:44.261986 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 59253.7us
W20240426 09:55:44.285444 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 63931.9us
I20240426 09:55:44.289886 10084 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:44.290516 10084 raft_consensus.cc:595] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
W20240426 09:55:44.318141 12367 log.cc:924] Time spent T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20240426 09:55:44.328583 12332 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:44.333040 12375 ts_tablet_manager.cc:902] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:44.330569 12332 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:44.345369 9835 catalog_manager.cc:5484] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index -1 to 5, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 6 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:44.342461 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:44.343233 9939 raft_consensus.cc:2932] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:44.379017 10082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:44.379702 10082 raft_consensus.cc:595] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:44.405673 10081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:44.406255 10081 raft_consensus.cc:595] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:44.407984 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:44.409250 10014 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:44.413146 10013 raft_consensus.cc:2932] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:44.414055 10082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:44.414664 10082 raft_consensus.cc:595] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:44.413040 12370 raft_consensus.cc:979] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:44.416297 12370 raft_consensus.cc:1067] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:44.418097 10081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:44.419229 10081 raft_consensus.cc:595] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:44.427369 10014 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:55:44.431918 10015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:55:44.432512 10015 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:44.432901 10015 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:44.448129 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 09:55:44.439942 10055 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:55:44.449790 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:55:44.450439 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:44.454188 12370 raft_consensus.cc:979] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:44.455477 12377 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:44.455996 12377 raft_consensus.cc:686] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:44.456758 12377 consensus_queue.cc:237] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:44.454679 12370 raft_consensus.cc:1067] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
W20240426 09:55:44.494050 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 53445.2us
I20240426 09:55:44.496642 10009 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:55:44.497184 10009 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:44.497540 10009 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:44.507328 10009 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:44.500053 10015 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:44.510915 10009 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:44.517457 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:44.518168 10082 raft_consensus.cc:3032] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:44.518528 10082 raft_consensus.cc:729] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:44.521010 10015 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:44.521400 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:44.522007 10089 raft_consensus.cc:3032] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:55:44.522269 10089 raft_consensus.cc:729] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:44.522958 10089 consensus_queue.cc:260] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:44.523983 10089 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:44.519317 10082 consensus_queue.cc:260] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:44.531961 10089 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:55:44.538437 9979 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:44.548255 10082 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:44.557735 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:44.558358 9935 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:44.559389 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:44.559958 9938 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:44.549472 12359 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:55:44.571596 10082 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:44.572075 9938 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:44.572518 9979 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:44.597301 12217 raft_consensus.cc:2781] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:44.591697 12359 raft_consensus.cc:686] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:44.649276 12242 raft_consensus.cc:979] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:44.649551 12359 consensus_queue.cc:237] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:44.659178 9935 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:55:44.669260 12242 raft_consensus.cc:1067] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:44.679122 10009 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:55:44.679725 10009 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:44.680070 10009 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:44.680492 12369 raft_consensus.cc:979] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:44.680974 12369 raft_consensus.cc:1067] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:44.690022 9835 catalog_manager.cc:5484] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 6 to 7, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 7 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:44.700842 10013 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:55:44.701457 10013 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:44.701807 10013 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:44.712942 10013 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:44.714748 10009 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:44.716111 10009 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:44.717195 10013 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:55:44.723104 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 110565us
I20240426 09:55:44.726011 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:44.726704 10081 raft_consensus.cc:3032] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:55:44.726994 10081 raft_consensus.cc:729] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:44.727701 10081 consensus_queue.cc:260] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:44.728876 10081 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:44.735097 12217 raft_consensus.cc:686] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:44.725417 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:44.736968 10082 raft_consensus.cc:3032] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:55:44.737282 10082 raft_consensus.cc:729] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:44.737993 10082 consensus_queue.cc:260] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:44.740070 10082 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:44.744457 10081 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:55:44.747973 12392 ts_tablet_manager.cc:902] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:44.756357 9979 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
W20240426 09:55:44.768443 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 150389us
I20240426 09:55:44.771314 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:44.767002 10082 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:55:44.771899 9935 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:44.771624 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:44.772632 9937 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:44.776885 9937 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:55:44.776961 9935 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:55:44.770330 12391 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:44.824780 10089 raft_consensus.cc:1226] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7. Ops: []
I20240426 09:55:44.843489 12265 raft_consensus.cc:979] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee: : Instructing follower f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:44.843979 12265 raft_consensus.cc:1067] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Signalling peer f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:44.849006 12382 consensus_queue.cc:1046] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:44.849758 12382 consensus_queue.cc:1225] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer d3a480e831d649139ff016fb68aee3ee log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240426 09:55:44.850979 12382 raft_consensus.cc:3032] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:55:44.851308 12382 raft_consensus.cc:729] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:44.852034 12382 consensus_queue.cc:260] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:44.818190 12391 raft_consensus.cc:686] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:44.853861 9935 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:47248
I20240426 09:55:44.854395 9935 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:44.854732 9935 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:44.817590 12375 tablet_copy_client.cc:308] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.130:44411
I20240426 09:55:44.863490 9935 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:44.869992 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 49391.5us
I20240426 09:55:44.787436 12217 consensus_queue.cc:237] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:44.842639 9839 catalog_manager.cc:5484] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 7 to 8, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 8 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:44.877137 9935 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:44.881062 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:44.879693 12402 ts_tablet_manager.cc:902] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:44.876371 12382 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
W20240426 09:55:44.898726 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 130221us
I20240426 09:55:44.907927 9982 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:44.918179 12206 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:55:44.918535 10026 tablet_copy_service.cc:140] P 826f03b2fb9c4b9483623b041a1bf80e: Received BeginTabletCopySession request for tablet 1c0134eb2ced4f5c8e56412aaee1c225 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:36762)
I20240426 09:55:44.919203 10026 tablet_copy_service.cc:161] P 826f03b2fb9c4b9483623b041a1bf80e: Beginning new tablet copy session on tablet 1c0134eb2ced4f5c8e56412aaee1c225 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:36762: session id = 1fee3da01ebc463396397057fdb81a68-1c0134eb2ced4f5c8e56412aaee1c225
I20240426 09:55:44.918705 12206 raft_consensus.cc:686] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:44.920425 12206 consensus_queue.cc:237] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:44.900898 12365 tablet_copy_client.cc:308] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.131:34821
I20240426 09:55:44.908682 12391 consensus_queue.cc:237] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:44.965274 10100 tablet_copy_service.cc:140] P d3a480e831d649139ff016fb68aee3ee: Received BeginTabletCopySession request for tablet 472a3b5debc94412b63080f474ae4fed from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:48202)
I20240426 09:55:44.965891 10100 tablet_copy_service.cc:161] P d3a480e831d649139ff016fb68aee3ee: Beginning new tablet copy session on tablet 472a3b5debc94412b63080f474ae4fed from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:48202: session id = 1fee3da01ebc463396397057fdb81a68-472a3b5debc94412b63080f474ae4fed
I20240426 09:55:44.983197 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:44.988001 10014 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:44.983510 10089 raft_consensus.cc:3032] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:44.988556 10089 raft_consensus.cc:729] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:44.989251 10089 consensus_queue.cc:260] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:44.994602 10015 raft_consensus.cc:1226] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7. Ops: []
I20240426 09:55:45.007256 10089 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:45.015893 9937 raft_consensus.cc:1261] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 09:55:45.024840 10089 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:45.023728 10014 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:45.062574 10026 tablet_copy_source_session.cc:205] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:45.070675 10011 raft_consensus.cc:1261] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 09:55:45.073685 12375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0134eb2ced4f5c8e56412aaee1c225. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:45.055013 12369 consensus_queue.cc:1035] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.024s
I20240426 09:55:45.079686 12369 consensus_queue.cc:1225] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer f12521b6d51242629cf4297864df00b9 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240426 09:55:45.066380 10100 tablet_copy_source_session.cc:205] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:45.099449 9906 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:45.101308 9839 catalog_manager.cc:5484] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 5 to 6, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 6 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:45.092446 12369 consensus_queue.cc:1035] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.018s
I20240426 09:55:45.109025 9839 catalog_manager.cc:5484] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 6 to 7, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 7 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:45.114066 9839 catalog_manager.cc:5484] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 7 to 8, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 8 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:45.118968 9938 pending_rounds.cc:76] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9: Aborting all ops after (but not including) 2
I20240426 09:55:45.119313 9938 pending_rounds.cc:97] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20240426 09:55:45.119612 9938 raft_consensus.cc:2866] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:55:45.142975 12354 raft_consensus.cc:2781] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:45.202680 9937 raft_consensus.cc:1261] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 09:55:45.170610 12354 raft_consensus.cc:686] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:45.215358 12365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472a3b5debc94412b63080f474ae4fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:45.231006 12354 consensus_queue.cc:237] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:45.237262 12217 consensus_queue.cc:1035] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20240426 09:55:45.254559 12402 tablet_copy_client.cc:308] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.131:34821
I20240426 09:55:45.265172 12392 tablet_copy_client.cc:308] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.130:44411
I20240426 09:55:45.306329 9835 catalog_manager.cc:5484] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 4 to 5, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 5 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:45.323477 10026 tablet_copy_service.cc:140] P 826f03b2fb9c4b9483623b041a1bf80e: Received BeginTabletCopySession request for tablet 67dfdb63e28a460486dfe163f03bf820 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:36762)
I20240426 09:55:45.324024 10026 tablet_copy_service.cc:161] P 826f03b2fb9c4b9483623b041a1bf80e: Beginning new tablet copy session on tablet 67dfdb63e28a460486dfe163f03bf820 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:36762: session id = 1fee3da01ebc463396397057fdb81a68-67dfdb63e28a460486dfe163f03bf820
I20240426 09:55:45.339390 10081 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 09:55:45.339713 10089 raft_consensus.cc:1261] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 09:55:45.339998 10100 tablet_copy_service.cc:140] P d3a480e831d649139ff016fb68aee3ee: Received BeginTabletCopySession request for tablet 308d6ed4b68346be85caf2871916eab9 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:48202)
I20240426 09:55:45.345003 12217 consensus_queue.cc:1035] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:45.345373 10100 tablet_copy_service.cc:161] P d3a480e831d649139ff016fb68aee3ee: Beginning new tablet copy session on tablet 308d6ed4b68346be85caf2871916eab9 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:48202: session id = 1fee3da01ebc463396397057fdb81a68-308d6ed4b68346be85caf2871916eab9
I20240426 09:55:45.356832 12273 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:45.359660 10026 tablet_copy_source_session.cc:205] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:45.365409 9939 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 09:55:45.367808 12392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67dfdb63e28a460486dfe163f03bf820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:45.377055 10100 tablet_copy_source_session.cc:205] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:45.371760 12273 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20240426 09:55:45.394220 12402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 308d6ed4b68346be85caf2871916eab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:45.409981 12365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:45570 (request call id 229) took 1247 ms. Trace:
I20240426 09:55:45.410384 12365 rpcz_store.cc:276] 0426 09:55:44.162563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:44.162734 (+ 171us) service_pool.cc:225] Handling call
0426 09:55:44.265956 (+103222us) ts_tablet_manager.cc:903] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
0426 09:55:45.138779 (+872823us) tablet_copy_client.cc:353] Tablet copy session begun
0426 09:55:45.280788 (+142009us) tablet_metadata.cc:641] Metadata flushed
0426 09:55:45.281050 (+ 262us) tablet_copy_client.cc:482] Wrote new tablet metadata
0426 09:55:45.407201 (+126151us) tablet_copy_client.cc:403] Wrote new consensus metadata
0426 09:55:45.409587 (+ 2386us) tablet_replica.cc:205] Creating consensus instance
0426 09:55:45.409846 (+ 259us) ts_tablet_manager.cc:958] Replica registered.
0426 09:55:45.409913 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":26512,"tablet-copy.queue_time_us":71945,"thread_start_us":631741,"threads_started":5}
I20240426 09:55:45.411813 12365 tablet_copy_client.cc:783] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:45.412312 12365 tablet_copy_client.cc:647] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:45.412750 12375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:45564 (request call id 317) took 1145 ms. Trace:
I20240426 09:55:45.413127 12375 rpcz_store.cc:276] 0426 09:55:44.258038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:55:44.258294 (+ 256us) service_pool.cc:225] Handling call
0426 09:55:44.333389 (+ 75095us) ts_tablet_manager.cc:903] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
0426 09:55:45.072500 (+739111us) tablet_copy_client.cc:353] Tablet copy session begun
0426 09:55:45.289115 (+216615us) tablet_metadata.cc:641] Metadata flushed
0426 09:55:45.289361 (+ 246us) tablet_copy_client.cc:482] Wrote new tablet metadata
0426 09:55:45.334641 (+ 45280us) tablet_copy_client.cc:403] Wrote new consensus metadata
0426 09:55:45.374915 (+ 40274us) tablet_replica.cc:205] Creating consensus instance
0426 09:55:45.402748 (+ 27833us) ts_tablet_manager.cc:958] Replica registered.
0426 09:55:45.403451 (+ 703us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":28027,"tablet-copy.queue_time_us":74241,"thread_start_us":419481,"threads_started":5}
I20240426 09:55:45.414371 12375 tablet_copy_client.cc:783] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:45.428740 12375 tablet_copy_client.cc:647] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:45.439512 12375 tablet_copy_client.cc:523] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:45.434363 12365 tablet_copy_client.cc:523] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:45.461261 12365 tablet_bootstrap.cc:492] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:45.487367 12392 tablet_copy_client.cc:783] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:45.497061 12392 tablet_copy_client.cc:647] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:45.503540 10086 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 09:55:45.506286 12359 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:45.508159 12375 tablet_bootstrap.cc:492] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
W20240426 09:55:45.510596 9980 consensus_peers.cc:485] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 308d6ed4b68346be85caf2871916eab9. This is attempt 1: this message will repeat every 5th retry.
W20240426 09:55:45.527271 12426 log.cc:924] Time spent T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20240426 09:55:45.548799 9935 raft_consensus.cc:1261] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:55:45.575575 12217 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:45.575299 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:45.591780 10081 raft_consensus.cc:1261] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:55:45.556948 12392 tablet_copy_client.cc:523] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:45.611196 12402 tablet_copy_client.cc:783] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:45.611807 12402 tablet_copy_client.cc:647] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:45.619395 12402 tablet_copy_client.cc:523] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:45.623519 9937 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 09:55:45.623135 12217 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:45.653795 10084 raft_consensus.cc:1261] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:45.671736 12355 consensus_queue.cc:1035] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20240426 09:55:45.683013 10084 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:45.683658 10084 raft_consensus.cc:595] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:45.658490 12399 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20240426 09:55:45.755425 10081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:45.756047 10081 raft_consensus.cc:595] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:45.772195 12392 tablet_bootstrap.cc:492] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:45.774037 10086 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:45.774629 10086 raft_consensus.cc:595] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:45.785432 12402 tablet_bootstrap.cc:492] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
W20240426 09:55:45.832224 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 80327.6us
I20240426 09:55:45.838968 10016 raft_consensus.cc:1261] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:45.839767 10015 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:45.840381 10015 raft_consensus.cc:595] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
W20240426 09:55:45.843675 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 81962us
I20240426 09:55:45.853480 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 09:55:45.854671 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:55:45.855194 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:45.858233 12354 consensus_queue.cc:1035] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20240426 09:55:45.887190 12369 raft_consensus.cc:979] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee: : Instructing follower f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:45.887688 12369 raft_consensus.cc:1067] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Signalling peer f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:45.891225 12206 raft_consensus.cc:979] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:45.891667 12206 raft_consensus.cc:1067] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Signalling peer f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:45.894742 9933 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:47242
I20240426 09:55:45.895345 9933 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:45.895669 9933 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:45.911500 9935 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:47248
I20240426 09:55:45.913120 9933 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:45.914620 9938 raft_consensus.cc:1226] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 6. Current term is 7. Ops: []
I20240426 09:55:45.912153 9935 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:45.915338 9935 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:45.917769 12436 consensus_queue.cc:1046] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:45.918805 12394 raft_consensus.cc:3032] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:55:45.919159 12394 raft_consensus.cc:729] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:45.920166 12365 tablet_bootstrap.cc:492] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:45.919924 12394 consensus_queue.cc:260] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:45.921082 12365 tablet_bootstrap.cc:492] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:45.921583 9933 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:45.921890 12365 ts_tablet_manager.cc:1367] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.461s user 0.034s sys 0.021s
I20240426 09:55:45.922539 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:45.921099 12394 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:45.923450 10089 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:45.932363 12375 tablet_bootstrap.cc:492] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:45.933297 12375 tablet_bootstrap.cc:492] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:45.934013 12375 ts_tablet_manager.cc:1367] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.426s user 0.035s sys 0.007s
I20240426 09:55:45.939319 9935 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:45.944196 10089 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:55:45.949746 9904 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:45.956177 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:45.951139 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:45.956880 10081 raft_consensus.cc:3032] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:55:45.957199 10081 raft_consensus.cc:729] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:45.957886 10081 consensus_queue.cc:260] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:45.959062 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:45.959662 10009 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:45.962582 10081 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:45.963806 12435 raft_consensus.cc:2781] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:45.964313 12435 raft_consensus.cc:686] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:45.971264 10016 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:55:45.972198 12365 raft_consensus.cc:348] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:45.946128 9935 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:45.978196 10009 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:55:45.968672 12435 consensus_queue.cc:237] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:46.021539 12365 raft_consensus.cc:729] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:45.987215 10081 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:55:45.968518 12375 raft_consensus.cc:348] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:46.023360 12375 raft_consensus.cc:729] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:46.024016 12375 consensus_queue.cc:260] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:46.026459 12375 ts_tablet_manager.cc:1398] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.092s user 0.006s sys 0.000s
I20240426 09:55:46.022275 12365 consensus_queue.cc:260] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:46.029443 12365 ts_tablet_manager.cc:1398] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.062s user 0.005s sys 0.000s
I20240426 09:55:46.031322 10026 tablet_copy_service.cc:342] P 826f03b2fb9c4b9483623b041a1bf80e: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-1c0134eb2ced4f5c8e56412aaee1c225 received from {username='slave'} at 127.0.0.1:36762
I20240426 09:55:46.031525 9904 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:46.031787 10026 tablet_copy_service.cc:433] P 826f03b2fb9c4b9483623b041a1bf80e: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-1c0134eb2ced4f5c8e56412aaee1c225 on tablet 1c0134eb2ced4f5c8e56412aaee1c225 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:46.043877 10100 tablet_copy_service.cc:342] P d3a480e831d649139ff016fb68aee3ee: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-472a3b5debc94412b63080f474ae4fed received from {username='slave'} at 127.0.0.1:48202
I20240426 09:55:46.044320 10100 tablet_copy_service.cc:433] P d3a480e831d649139ff016fb68aee3ee: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-472a3b5debc94412b63080f474ae4fed on tablet 472a3b5debc94412b63080f474ae4fed with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:46.072065 12357 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:46.072629 12357 raft_consensus.cc:686] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:46.073422 12357 consensus_queue.cc:237] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:46.087929 9838 catalog_manager.cc:5484] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 6 to 7, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 7 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:46.095865 12402 tablet_bootstrap.cc:492] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:46.105983 12438 raft_consensus.cc:979] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:46.106559 12438 raft_consensus.cc:1067] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:46.099341 12402 tablet_bootstrap.cc:492] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:46.117672 10016 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:55:46.118227 10016 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:46.119491 12402 ts_tablet_manager.cc:1367] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.367s user 0.026s sys 0.002s
I20240426 09:55:46.123371 12392 tablet_bootstrap.cc:492] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:46.141196 10016 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:46.144726 12402 raft_consensus.cc:348] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:46.146878 12402 raft_consensus.cc:729] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:46.147809 12392 tablet_bootstrap.cc:492] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:46.147600 12402 consensus_queue.cc:260] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:46.148622 12392 ts_tablet_manager.cc:1367] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.377s user 0.025s sys 0.006s
I20240426 09:55:46.159950 12402 ts_tablet_manager.cc:1398] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20240426 09:55:46.160845 12392 raft_consensus.cc:348] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:46.161746 12392 raft_consensus.cc:729] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:46.162364 12392 consensus_queue.cc:260] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:46.166751 10100 tablet_copy_service.cc:342] P d3a480e831d649139ff016fb68aee3ee: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-308d6ed4b68346be85caf2871916eab9 received from {username='slave'} at 127.0.0.1:48202
I20240426 09:55:46.164745 12392 ts_tablet_manager.cc:1398] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20240426 09:55:46.167232 10100 tablet_copy_service.cc:433] P d3a480e831d649139ff016fb68aee3ee: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-308d6ed4b68346be85caf2871916eab9 on tablet 308d6ed4b68346be85caf2871916eab9 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:46.170886 10026 tablet_copy_service.cc:342] P 826f03b2fb9c4b9483623b041a1bf80e: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-67dfdb63e28a460486dfe163f03bf820 received from {username='slave'} at 127.0.0.1:36762
I20240426 09:55:46.171288 10026 tablet_copy_service.cc:433] P 826f03b2fb9c4b9483623b041a1bf80e: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-67dfdb63e28a460486dfe163f03bf820 on tablet 67dfdb63e28a460486dfe163f03bf820 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:46.173679 10016 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:46.178934 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:46.179471 9935 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:46.187778 10613 raft_consensus.cc:1203] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Deduplicated request from leader. Original: 7.4->[7.5-7.5] Dedup: 7.5->[]
I20240426 09:55:46.194418 10016 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:46.201202 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:46.201611 10081 raft_consensus.cc:3032] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:46.201797 10081 raft_consensus.cc:729] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:46.202306 10081 consensus_queue.cc:260] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:46.203018 10081 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:46.204519 9935 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:46.205693 9982 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:46.210611 10081 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:46.209956 12436 raft_consensus.cc:2781] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:46.213536 12436 raft_consensus.cc:686] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:46.224179 12369 raft_consensus.cc:979] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:46.222353 12436 consensus_queue.cc:237] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:46.228211 12242 raft_consensus.cc:1067] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:46.229626 10016 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:55:46.230157 10016 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:46.231544 10016 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:46.244165 10016 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:46.246316 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:46.246897 9935 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:46.253796 9935 raft_consensus.cc:2454] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:46.254900 10016 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:46.256357 9982 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:46.256421 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:46.259223 10081 raft_consensus.cc:3032] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:46.259532 10081 raft_consensus.cc:729] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:46.260248 10081 consensus_queue.cc:260] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:46.261484 10081 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:46.263458 12436 raft_consensus.cc:2781] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:46.263948 12436 raft_consensus.cc:686] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:46.271091 10081 raft_consensus.cc:2454] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:55:46.273450 12436 consensus_queue.cc:237] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:46.339610 9839 catalog_manager.cc:5484] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 6 to 7, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 7 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:46.356005 9838 catalog_manager.cc:5484] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 5 to 6, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 6 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:46.357581 9838 catalog_manager.cc:5484] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 4 to 5, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 5 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:46.471146 10081 raft_consensus.cc:1261] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:55:46.473233 12354 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:46.491092 10015 raft_consensus.cc:1261] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:55:46.492731 12357 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:46.501296 10613 raft_consensus.cc:1203] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20240426 09:55:46.581996 12438 raft_consensus.cc:1050] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:46.584542 12438 consensus_queue.cc:237] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 3, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:46.622102 10613 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 09:55:46.624290 12369 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:46.626516 9935 raft_consensus.cc:1261] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:55:46.630970 9933 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 09:55:46.633252 12359 consensus_queue.cc:1035] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:46.636698 12242 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:46.645377 10014 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 09:55:46.647954 12370 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:46.651787 10089 raft_consensus.cc:1261] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:55:46.653127 12357 consensus_queue.cc:1035] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:46.666817 10612 raft_consensus.cc:1203] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.3->[4.4-4.4] Dedup: 4.4->[]
I20240426 09:55:46.682761 10015 raft_consensus.cc:1261] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:55:46.691320 12356 consensus_queue.cc:1035] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:46.721801 12242 raft_consensus.cc:2932] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Committing config change with OpId 7.6: config changed from index 5 to 6, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:46.726505 10081 raft_consensus.cc:1261] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:55:46.729045 12359 consensus_queue.cc:1035] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:46.772440 9933 raft_consensus.cc:2932] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index 5 to 6, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:46.728164 10613 raft_consensus.cc:2932] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index 5 to 6, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:46.834300 9835 catalog_manager.cc:5484] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 5 to 6, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:46.846318 9935 raft_consensus.cc:1261] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240426 09:55:46.862846 12359 consensus_queue.cc:1035] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:46.894394 10084 consensus_queue.cc:237] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:46.906844 10613 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:55:46.908495 12370 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:46.915885 10014 raft_consensus.cc:2932] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index 5 to 6, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:46.915227 12438 raft_consensus.cc:2932] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Committing config change with OpId 7.7: config changed from index 6 to 7, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:46.925698 12359 raft_consensus.cc:1050] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:46.927317 10081 raft_consensus.cc:1261] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240426 09:55:46.928402 12359 consensus_queue.cc:237] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:46.931536 12440 raft_consensus.cc:1050] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:46.932639 10612 raft_consensus.cc:2932] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 6 to 7, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:46.933801 12440 consensus_queue.cc:237] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:46.941076 9935 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:55:46.947233 12436 consensus_queue.cc:1035] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:46.973275 12369 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:46.990348 9836 catalog_manager.cc:5484] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 6 to 7, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New cstate: current_term: 7 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:46.992050 9823 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 472a3b5debc94412b63080f474ae4fed with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:55:47.002622 10006 tablet_service.cc:1514] Processing DeleteTablet for tablet 472a3b5debc94412b63080f474ae4fed with delete_type TABLET_DATA_TOMBSTONED (TS 826f03b2fb9c4b9483623b041a1bf80e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34362
I20240426 09:55:47.007251 9935 raft_consensus.cc:2932] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 6 to 7, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
W20240426 09:55:47.097572 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 80383.7us
I20240426 09:55:47.117601 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:47.135957 12470 tablet_replica.cc:330] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e: stopping tablet replica
I20240426 09:55:47.147672 10612 raft_consensus.cc:1261] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEARNER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240426 09:55:47.151521 12465 consensus_queue.cc:1035] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:47.164484 10613 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:55:47.164876 9938 raft_consensus.cc:1261] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240426 09:55:47.165859 12465 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:47.195577 10089 raft_consensus.cc:1261] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240426 09:55:47.198668 10081 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:55:47.202373 12470 raft_consensus.cc:2229] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Raft consensus shutting down.
I20240426 09:55:47.223809 9933 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20240426 09:55:47.227558 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 108374us
I20240426 09:55:47.249009 12465 consensus_queue.cc:1035] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240426 09:55:47.250149 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 48596.3us
I20240426 09:55:47.262420 12440 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:47.263881 12470 raft_consensus.cc:2258] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Raft consensus is shut down!
W20240426 09:55:47.264478 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 50078.2us
I20240426 09:55:47.293390 12470 ts_tablet_manager.cc:1875] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:55:47.308151 10015 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:34938
W20240426 09:55:47.334285 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 130177us
I20240426 09:55:47.334975 10015 raft_consensus.cc:595] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Received request to transfer leadership to TS d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:47.324806 12470 ts_tablet_manager.cc:1888] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.6
I20240426 09:55:47.339818 12470 log.cc:1196] T 472a3b5debc94412b63080f474ae4fed P 826f03b2fb9c4b9483623b041a1bf80e: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-1-root/wals/472a3b5debc94412b63080f474ae4fed
I20240426 09:55:47.342301 12436 consensus_queue.cc:1035] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:47.383917 12462 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 09:55:47.393028 12478 raft_consensus.cc:1050] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:47.420352 12483 raft_consensus.cc:979] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:47.421068 12483 raft_consensus.cc:1067] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Signalling peer d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:47.426589 10081 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:59746
I20240426 09:55:47.427165 10081 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:47.427515 10081 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:47.438977 10081 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:47.440222 10089 raft_consensus.cc:1226] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8. Ops: []
I20240426 09:55:47.442817 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:47.443374 10613 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:47.442138 12478 consensus_queue.cc:237] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 2, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:47.449013 10081 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:47.449065 10613 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:55:47.452134 12462 raft_consensus.cc:2932] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:47.450246 12477 consensus_queue.cc:1046] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:47.460036 10084 raft_consensus.cc:2932] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:47.429072 12483 raft_consensus.cc:2932] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Committing config change with OpId 7.7: config changed from index 5 to 7, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:47.489032 9838 catalog_manager.cc:5484] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index 5 to 7, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:47.484114 12483 raft_consensus.cc:3032] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:55:47.492401 12483 raft_consensus.cc:729] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:47.493206 12483 consensus_queue.cc:260] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:47.495558 12483 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:47.514611 9838 catalog_manager.cc:5484] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index 3 to 4, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:55:47.538142 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:47.538800 9938 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:47.540217 9825 catalog_manager.cc:4830] TS 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): tablet 472a3b5debc94412b63080f474ae4fed (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:55:47.541325 10613 raft_consensus.cc:1226] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8. Ops: []
I20240426 09:55:47.541082 10612 raft_consensus.cc:2932] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:47.565919 9938 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:55:47.568243 9933 raft_consensus.cc:1226] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20240426 09:55:47.568995 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:47.574258 10056 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:47.577919 12438 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:55:47.578380 12438 raft_consensus.cc:686] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:47.593984 10009 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:55:47.609133 9933 raft_consensus.cc:1261] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240426 09:55:47.609824 10084 raft_consensus.cc:1261] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240426 09:55:47.610884 10618 raft_consensus.cc:1261] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 4 LEARNER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240426 09:55:47.616137 12465 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:47.618147 12438 consensus_queue.cc:237] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:47.633864 12465 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:47.643599 9937 raft_consensus.cc:2932] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:47.654733 9839 catalog_manager.cc:5484] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index 7 to 5, term changed from 7 to 8, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from VOTER to NON_VOTER, now has a pending config: VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129), VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130), VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131), VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 8 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:47.667356 9937 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:46580
I20240426 09:55:47.667946 9937 raft_consensus.cc:595] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:47.677860 10016 consensus_queue.cc:237] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:47.681383 10015 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:47.681955 10015 raft_consensus.cc:595] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Received request to transfer leadership to TS d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:47.684168 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240426 09:55:47.685868 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:55:47.686519 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240426 09:55:47.688823 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 115178us
I20240426 09:55:47.697825 10612 raft_consensus.cc:1261] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 09:55:47.698482 12479 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:47.700541 9933 raft_consensus.cc:1261] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20240426 09:55:47.706058 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 131869us
I20240426 09:55:47.708230 12465 consensus_queue.cc:1035] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20240426 09:55:47.727145 12440 consensus_queue.cc:1035] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:47.794515 10613 raft_consensus.cc:2932] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:47.798041 12440 raft_consensus.cc:2932] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:47.801915 12483 raft_consensus.cc:979] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:47.802414 12483 raft_consensus.cc:1067] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Signalling peer d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:47.805123 9935 raft_consensus.cc:2932] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:47.824262 9839 catalog_manager.cc:5484] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 4 to 5, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New cstate: current_term: 3 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:47.833034 12479 raft_consensus.cc:2932] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:47.838234 10077 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c0134eb2ced4f5c8e56412aaee1c225 with delete_type TABLET_DATA_TOMBSTONED (TS d3a480e831d649139ff016fb68aee3ee not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59730
I20240426 09:55:47.837915 9935 raft_consensus.cc:2932] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:47.850982 10084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:59746
I20240426 09:55:47.851601 10084 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:47.851958 10084 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:47.859627 12497 tablet_replica.cc:330] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee: stopping tablet replica
I20240426 09:55:47.860642 12497 raft_consensus.cc:2229] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 09:55:47.861271 10084 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:47.864464 10084 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:47.861341 12497 raft_consensus.cc:2258] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 09:55:47.870335 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:47.871176 10016 raft_consensus.cc:3032] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:47.869668 10618 raft_consensus.cc:2932] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:47.871745 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:47.872164 9935 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:47.871470 10016 raft_consensus.cc:729] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:47.878172 9935 raft_consensus.cc:2454] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:47.877656 10016 consensus_queue.cc:260] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:47.879782 10056 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:47.879746 10016 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:47.882212 12370 raft_consensus.cc:2781] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:47.884233 12370 raft_consensus.cc:686] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:47.898334 10016 raft_consensus.cc:2454] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:47.901635 9839 catalog_manager.cc:5484] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 reported cstate change: config changed from index 4 to 5, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:47.907260 12497 ts_tablet_manager.cc:1875] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:55:47.885033 12370 consensus_queue.cc:237] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:47.933228 12491 raft_consensus.cc:979] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:47.937363 12357 raft_consensus.cc:1067] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:47.939329 10014 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34390
I20240426 09:55:47.939901 10014 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:47.940270 10014 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:47.942102 9836 catalog_manager.cc:5484] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 5 to 6, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 6 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:47.948163 9825 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c0134eb2ced4f5c8e56412aaee1c225 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240426 09:55:47.939363 12496 log.cc:924] Time spent T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.114s user 0.003s sys 0.000s
I20240426 09:55:47.957033 10089 raft_consensus.cc:2932] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:47.970285 10014 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:47.972404 10014 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:47.974357 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:47.974973 10084 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:47.987696 10084 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:55:47.988512 9979 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:47.989140 12440 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:55:47.989580 12440 raft_consensus.cc:686] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:47.991711 12440 consensus_queue.cc:237] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:48.001616 12497 ts_tablet_manager.cc:1888] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20240426 09:55:48.012524 12497 log.cc:1196] T 1c0134eb2ced4f5c8e56412aaee1c225 P d3a480e831d649139ff016fb68aee3ee: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-2-root/wals/1c0134eb2ced4f5c8e56412aaee1c225
I20240426 09:55:48.014289 9823 catalog_manager.cc:4830] TS d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): tablet 1c0134eb2ced4f5c8e56412aaee1c225 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:55:48.018920 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:48.019527 9932 raft_consensus.cc:3032] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:55:48.019855 9932 raft_consensus.cc:729] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:48.020546 9932 consensus_queue.cc:260] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:48.021664 9932 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:48.035821 9932 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:55:48.067909 9836 catalog_manager.cc:5484] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 7 to 8, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 8 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:48.090435 10014 consensus_queue.cc:237] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:48.097581 10618 raft_consensus.cc:1261] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 09:55:48.097667 10089 raft_consensus.cc:1261] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 09:55:48.099030 12440 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:48.100311 12465 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20240426 09:55:48.116297 12465 raft_consensus.cc:2932] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:48.119943 10084 raft_consensus.cc:2932] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:48.122143 10612 raft_consensus.cc:2932] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:48.132792 9825 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 67dfdb63e28a460486dfe163f03bf820 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:55:48.133615 9932 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 09:55:48.148849 9836 catalog_manager.cc:5484] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 5 to 6, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New cstate: current_term: 4 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:48.150820 10618 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 09:55:48.149458 12370 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:48.152854 12438 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:48.167606 9925 tablet_service.cc:1514] Processing DeleteTablet for tablet 67dfdb63e28a460486dfe163f03bf820 with delete_type TABLET_DATA_TOMBSTONED (TS f12521b6d51242629cf4297864df00b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47232
I20240426 09:55:48.180902 12500 tablet_replica.cc:330] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9: stopping tablet replica
I20240426 09:55:48.188133 12500 raft_consensus.cc:2229] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 09:55:48.188920 12500 raft_consensus.cc:2258] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 09:55:48.192243 12500 ts_tablet_manager.cc:1875] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:55:48.224802 12370 raft_consensus.cc:2932] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Committing config change with OpId 7.7: config changed from index 5 to 7, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:48.227516 12500 ts_tablet_manager.cc:1888] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.5
I20240426 09:55:48.229290 12500 log.cc:1196] T 67dfdb63e28a460486dfe163f03bf820 P f12521b6d51242629cf4297864df00b9: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-0-root/wals/67dfdb63e28a460486dfe163f03bf820
I20240426 09:55:48.228950 9932 raft_consensus.cc:2932] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 5 to 7, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:48.236050 9826 catalog_manager.cc:4830] TS f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): tablet 67dfdb63e28a460486dfe163f03bf820 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:55:48.244923 10009 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 09:55:48.246851 12438 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 09:55:48.287050 9836 catalog_manager.cc:5484] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 reported cstate change: config changed from index 5 to 7, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129), VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130), VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131), VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 8 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:48.322474 10612 raft_consensus.cc:2932] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 5 to 7, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:48.330555 10016 raft_consensus.cc:1261] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240426 09:55:48.332506 12438 consensus_queue.cc:1035] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:48.398931 9932 raft_consensus.cc:1261] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240426 09:55:48.400832 12493 consensus_queue.cc:1035] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:48.563287 10084 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240426 09:55:48.564742 12465 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:48.575136 9935 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240426 09:55:48.576747 12465 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:48.861030 10082 consensus_queue.cc:237] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:48.866888 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:48.904307 10618 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 09:55:48.905858 12438 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 09:55:48.915087 12493 raft_consensus.cc:2932] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Committing config change with OpId 8.9: config changed from index 7 to 9, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:48.916325 10618 raft_consensus.cc:2932] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Committing config change with OpId 8.9: config changed from index 7 to 9, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:48.926913 10009 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 09:55:48.929320 9823 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 308d6ed4b68346be85caf2871916eab9 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:55:48.932446 12370 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 09:55:48.937018 10009 raft_consensus.cc:2932] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Committing config change with OpId 8.9: config changed from index 7 to 9, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:48.970314 9839 catalog_manager.cc:5484] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index 7 to 9, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New cstate: current_term: 8 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 09:55:49.027917 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 50817.4us
I20240426 09:55:49.098897 9925 tablet_service.cc:1514] Processing DeleteTablet for tablet 308d6ed4b68346be85caf2871916eab9 with delete_type TABLET_DATA_TOMBSTONED (TS f12521b6d51242629cf4297864df00b9 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47232
I20240426 09:55:49.100126 10084 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:49.100716 10084 raft_consensus.cc:595] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:49.109738 10016 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:49.110318 10016 raft_consensus.cc:595] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:49.119087 10084 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:49.119695 10084 raft_consensus.cc:595] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:49.117022 12514 tablet_replica.cc:330] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9: stopping tablet replica
I20240426 09:55:49.121868 12514 raft_consensus.cc:2229] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Raft consensus shutting down.
I20240426 09:55:49.122797 12514 raft_consensus.cc:2258] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Raft consensus is shut down!
I20240426 09:55:49.127782 9935 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
mode: GRACEFUL
new_leader_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:46580
I20240426 09:55:49.128355 9935 raft_consensus.cc:595] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Received request to transfer leadership to TS d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:49.129554 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 09:55:49.131186 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:55:49.131762 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:49.135627 12514 ts_tablet_manager.cc:1875] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:55:49.164484 12514 ts_tablet_manager.cc:1888] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 8.8
I20240426 09:55:49.170064 12514 log.cc:1196] T 308d6ed4b68346be85caf2871916eab9 P f12521b6d51242629cf4297864df00b9: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-0-root/wals/308d6ed4b68346be85caf2871916eab9
I20240426 09:55:49.174888 9826 catalog_manager.cc:4830] TS f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): tablet 308d6ed4b68346be85caf2871916eab9 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:55:49.470316 12438 raft_consensus.cc:979] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:49.471383 12438 raft_consensus.cc:1067] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:49.473667 10016 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:55:49.474604 10016 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:49.475304 10016 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:55:49.495637 12510 raft_consensus.cc:979] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9: : Instructing follower d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:49.496271 12510 raft_consensus.cc:1067] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Signalling peer d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:49.501503 10081 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:59762
I20240426 09:55:49.502053 10081 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:49.502449 10081 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:49.536818 10081 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:49.553134 10016 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:49.555982 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:49.556730 10089 raft_consensus.cc:3032] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Stepping down as leader of term 8
I20240426 09:55:49.557016 10089 raft_consensus.cc:729] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:49.557768 10089 consensus_queue.cc:260] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:49.559839 10089 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:55:49.573704 10089 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 9.
I20240426 09:55:49.583487 10081 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:49.584295 10016 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:49.585801 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: true dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:49.591758 10613 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:55:49.595239 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:49.595831 9938 raft_consensus.cc:3032] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:49.596161 9938 raft_consensus.cc:729] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:49.596957 9938 consensus_queue.cc:260] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:49.601913 12479 raft_consensus.cc:979] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:49.602416 12479 raft_consensus.cc:1067] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Signalling peer f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:49.603334 10613 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 9.
I20240426 09:55:49.605587 9935 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:47242
I20240426 09:55:49.607394 9935 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:49.607899 9935 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:49.606564 9979 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:49.598613 9938 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:49.611250 12465 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election won for term 9
I20240426 09:55:49.611788 12465 raft_consensus.cc:686] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:49.612882 12465 consensus_queue.cc:237] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:49.631712 9938 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:49.633126 10056 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:49.631624 9935 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:49.635946 12493 raft_consensus.cc:2781] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:49.639504 12493 raft_consensus.cc:686] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:49.640437 12493 consensus_queue.cc:237] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:49.642571 9935 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:49.633416 9836 catalog_manager.cc:5484] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 8 to 9, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 9 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:49.647548 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:49.648110 10089 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:49.633141 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:49.649035 10014 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:49.650591 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:49.651185 10009 raft_consensus.cc:3032] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:49.651479 10009 raft_consensus.cc:729] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:49.652211 10009 consensus_queue.cc:260] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:49.653520 10009 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:49.668326 10014 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:49.676802 12520 raft_consensus.cc:979] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:49.677464 12519 raft_consensus.cc:1067] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:49.679535 10014 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:55:49.680044 10014 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:49.680351 10014 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:49.697537 10089 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:49.698697 9904 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:49.699581 12356 raft_consensus.cc:2781] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:49.699652 10009 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:55:49.700206 12356 raft_consensus.cc:686] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:49.701225 12356 consensus_queue.cc:237] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:49.706902 9838 catalog_manager.cc:5484] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 4 to 5, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 5 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:49.739650 10014 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:49.741559 10014 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:49.743650 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:49.744232 10089 raft_consensus.cc:3032] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:49.744488 10089 raft_consensus.cc:729] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:49.745186 10089 consensus_queue.cc:260] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:49.746390 10089 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:49.742650 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:49.747514 9935 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:49.773823 9839 catalog_manager.cc:5484] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 4 to 5, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 5 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:49.781831 9935 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:49.786907 9982 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:49.787875 12440 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:49.788357 12440 raft_consensus.cc:686] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:49.789219 12440 consensus_queue.cc:237] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:49.814612 10089 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:49.863435 9839 catalog_manager.cc:5484] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 5 to 6, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 6 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:50.024803 10612 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20240426 09:55:50.027040 12465 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 09:55:50.092077 10081 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20240426 09:55:50.093515 12479 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 09:55:50.145612 10014 raft_consensus.cc:1261] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 09:55:50.147079 12354 consensus_queue.cc:1035] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:50.174634 10081 consensus_queue.cc:237] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:50.190382 10089 raft_consensus.cc:1261] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 09:55:50.191942 12354 consensus_queue.cc:1035] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240426 09:55:50.221697 10054 consensus_peers.cc:485] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c75e7dd8bbe64a3eb2174cd4a218d51c. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:50.228466 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:50.229259 9935 raft_consensus.cc:1261] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:50.237998 10014 raft_consensus.cc:1261] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:55:50.242991 9935 raft_consensus.cc:1261] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:55:50.246110 12493 consensus_queue.cc:1035] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:50.246949 12519 consensus_queue.cc:1035] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:50.270476 9933 raft_consensus.cc:1261] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240426 09:55:50.271857 12440 consensus_queue.cc:1035] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:50.279080 10009 raft_consensus.cc:1261] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 09:55:50.256335 12520 consensus_queue.cc:1035] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20240426 09:55:50.295146 12493 consensus_queue.cc:1035] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 09:55:50.296572 10089 raft_consensus.cc:1261] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240426 09:55:50.301563 12440 consensus_queue.cc:1035] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:50.355437 12519 raft_consensus.cc:2932] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:50.359220 9937 raft_consensus.cc:2932] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:50.432674 10081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:50.433473 10081 raft_consensus.cc:595] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:50.468303 10009 raft_consensus.cc:2932] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:50.477660 9937 consensus_queue.cc:237] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:50.521876 10014 raft_consensus.cc:1261] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20240426 09:55:50.522388 9905 consensus_peers.cc:485] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb385fda44e94153ae3cd9ca21240939. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:50.523262 12354 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:50.524621 9937 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:46580
I20240426 09:55:50.525174 9937 raft_consensus.cc:595] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:50.540246 10089 raft_consensus.cc:1261] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 09:55:50.548635 9836 catalog_manager.cc:5484] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 reported cstate change: config changed from index -1 to 5, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 7 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:50.551348 10015 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:50.552012 10015 raft_consensus.cc:595] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:50.554718 10014 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:50.555290 10014 raft_consensus.cc:595] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Received request to transfer leadership to TS d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:50.556497 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 09:55:50.558135 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:55:50.562270 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:50.569483 12354 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20240426 09:55:50.596853 12354 raft_consensus.cc:2932] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Committing config change with OpId 7.6: config changed from index -1 to 6, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:50.599331 10089 raft_consensus.cc:2932] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index -1 to 6, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:50.605402 10015 raft_consensus.cc:2932] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index -1 to 6, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:50.606532 12510 raft_consensus.cc:979] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:50.607038 12510 raft_consensus.cc:1067] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:50.609269 10014 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34390
I20240426 09:55:50.609764 10014 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:50.610085 10014 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:50.616379 10014 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:50.620857 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:50.621390 9938 raft_consensus.cc:3032] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:50.621687 9938 raft_consensus.cc:729] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:50.621982 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:50.622525 10081 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:50.622361 9938 consensus_queue.cc:260] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:50.623430 9938 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:50.630008 10081 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:50.631112 9979 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:50.632947 12545 raft_consensus.cc:2781] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:50.633394 12545 raft_consensus.cc:686] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:50.634142 12545 consensus_queue.cc:237] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:50.630242 9938 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:50.644224 10014 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:50.681174 12493 raft_consensus.cc:979] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:50.681636 12493 raft_consensus.cc:1067] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:50.685266 10015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:55:50.685881 10015 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:50.686260 10015 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:50.699297 9839 catalog_manager.cc:5484] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 reported cstate change: config changed from index -1 to 6, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 7 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240426 09:55:50.705405 9887 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica f12521b6d51242629cf4297864df00b9 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:55:50.717639 10015 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:50.720530 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:50.720856 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: true dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:50.721421 10613 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:50.721925 9838 catalog_manager.cc:5484] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 5 to 6, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 6 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:50.721058 10089 raft_consensus.cc:3032] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:55:50.724614 10089 raft_consensus.cc:729] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:50.725425 10089 consensus_queue.cc:260] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:50.726440 10089 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
W20240426 09:55:50.728262 9887 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 693cab52ca1f45cab4af9282a6614f90, TS 826f03b2fb9c4b9483623b041a1bf80e -> TS 1fee3da01ebc463396397057fdb81a68 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:50.728611 9887 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 693cab52ca1f45cab4af9282a6614f90, TS 826f03b2fb9c4b9483623b041a1bf80e -> TS 1fee3da01ebc463396397057fdb81a68 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:50.733850 10089 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:55:50.739326 10613 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:55:50.748325 9980 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:50.749164 12479 raft_consensus.cc:2781] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:50.749608 12479 raft_consensus.cc:686] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:50.750453 12479 consensus_queue.cc:237] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:50.757254 10015 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:50.779359 9836 catalog_manager.cc:5484] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 6 to 7, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 7 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240426 09:55:50.757951 9887 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet fa1ede5c9b614a0f8b7adb20a7cfbbb1, TS f12521b6d51242629cf4297864df00b9 -> TS 1fee3da01ebc463396397057fdb81a68 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:50.797011 9887 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet fa1ede5c9b614a0f8b7adb20a7cfbbb1, TS f12521b6d51242629cf4297864df00b9 -> TS 1fee3da01ebc463396397057fdb81a68 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:50.803854 12558 ts_tablet_manager.cc:902] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:50.871698 12558 tablet_copy_client.cc:308] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.131:34821
I20240426 09:55:50.877930 10100 tablet_copy_service.cc:140] P d3a480e831d649139ff016fb68aee3ee: Received BeginTabletCopySession request for tablet c75e7dd8bbe64a3eb2174cd4a218d51c from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:48202)
I20240426 09:55:50.878496 10100 tablet_copy_service.cc:161] P d3a480e831d649139ff016fb68aee3ee: Beginning new tablet copy session on tablet c75e7dd8bbe64a3eb2174cd4a218d51c from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:48202: session id = 1fee3da01ebc463396397057fdb81a68-c75e7dd8bbe64a3eb2174cd4a218d51c
I20240426 09:55:50.880950 12545 raft_consensus.cc:979] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:50.881438 12545 raft_consensus.cc:1067] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Signalling peer d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:50.883056 10089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:59746
I20240426 09:55:50.883534 10089 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:50.883834 10089 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:50.889003 10100 tablet_copy_source_session.cc:205] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:50.915269 12440 raft_consensus.cc:979] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:50.915566 12558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c75e7dd8bbe64a3eb2174cd4a218d51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:50.915727 12440 raft_consensus.cc:1067] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Signalling peer f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:50.917559 9933 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:47242
I20240426 09:55:50.918072 9933 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:50.918391 9933 raft_consensus.cc:3037] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:50.922216 12563 ts_tablet_manager.cc:902] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113)
I20240426 09:55:50.940196 10089 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:50.940959 9933 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:50.950863 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:50.953886 9938 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:50.985311 12558 tablet_copy_client.cc:783] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:50.986377 12558 tablet_copy_client.cc:647] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:50.989230 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:50.989853 10613 raft_consensus.cc:3037] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:50.996654 10613 raft_consensus.cc:2454] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:50.998225 9905 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:50.999121 12356 raft_consensus.cc:2781] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:55:50.999641 12356 raft_consensus.cc:686] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [term 2 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:51.000571 12356 consensus_queue.cc:237] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:51.005326 12558 tablet_copy_client.cc:523] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:51.007947 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:51.016896 10014 raft_consensus.cc:3032] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 09:55:51.017177 10014 raft_consensus.cc:729] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:51.017983 10014 consensus_queue.cc:260] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:51.022724 10014 raft_consensus.cc:3037] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:55:51.027658 10089 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:51.028421 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:51.035537 10009 raft_consensus.cc:3032] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:55:51.035822 10009 raft_consensus.cc:729] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:51.036463 10009 consensus_queue.cc:260] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:51.037484 10009 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:51.031663 9933 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 2 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:51.045673 10082 raft_consensus.cc:1261] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240426 09:55:51.048183 10009 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:55:51.049510 10055 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:51.050293 12493 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:55:51.050818 12493 raft_consensus.cc:686] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:51.051950 12493 consensus_queue.cc:237] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:51.054215 12440 consensus_queue.cc:1035] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:51.083848 9836 catalog_manager.cc:5484] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 7 to 8, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 8 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:51.020682 12558 tablet_bootstrap.cc:492] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:51.113195 9938 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:55:51.128024 10014 raft_consensus.cc:2454] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 2.
I20240426 09:55:51.249176 12563 tablet_copy_client.cc:308] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.129:38113
W20240426 09:55:51.270660 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 45554.5us
I20240426 09:55:51.286897 9951 tablet_copy_service.cc:140] P f12521b6d51242629cf4297864df00b9: Received BeginTabletCopySession request for tablet fb385fda44e94153ae3cd9ca21240939 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:40928)
I20240426 09:55:51.287492 9951 tablet_copy_service.cc:161] P f12521b6d51242629cf4297864df00b9: Beginning new tablet copy session on tablet fb385fda44e94153ae3cd9ca21240939 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:40928: session id = 1fee3da01ebc463396397057fdb81a68-fb385fda44e94153ae3cd9ca21240939
W20240426 09:55:51.293828 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 98500.6us
I20240426 09:55:51.298676 9951 tablet_copy_source_session.cc:205] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:51.320183 9836 catalog_manager.cc:5484] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 1 to 2, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 2 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:51.321069 12563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb385fda44e94153ae3cd9ca21240939. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:51.332691 10089 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20240426 09:55:51.371148 12479 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
I20240426 09:55:51.382520 10613 raft_consensus.cc:1261] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:51.387866 10613 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20240426 09:55:51.399681 12572 consensus_queue.cc:1035] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20240426 09:55:51.407541 9935 raft_consensus.cc:1261] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240426 09:55:51.435182 12576 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20240426 09:55:51.450237 12511 consensus_queue.cc:1035] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 09:55:51.472429 12563 tablet_copy_client.cc:783] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:51.472993 12563 tablet_copy_client.cc:647] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
W20240426 09:55:51.491073 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 104845us
I20240426 09:55:51.500459 10015 raft_consensus.cc:1261] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 09:55:51.508813 12572 consensus_queue.cc:1035] T eb5d051be8c743a18cd550e145b6564e P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20240426 09:55:51.520355 12563 tablet_copy_client.cc:523] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240426 09:55:51.527577 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 85035.8us
I20240426 09:55:51.548097 12563 tablet_bootstrap.cc:492] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:51.560395 12558 tablet_bootstrap.cc:492] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:51.561467 12558 tablet_bootstrap.cc:492] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:51.562251 12558 ts_tablet_manager.cc:1367] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.542s user 0.041s sys 0.032s
I20240426 09:55:51.600811 12558 raft_consensus.cc:348] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:51.601708 12558 raft_consensus.cc:729] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:51.602396 12558 consensus_queue.cc:260] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:51.634243 12558 ts_tablet_manager.cc:1398] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.072s user 0.006s sys 0.002s
I20240426 09:55:51.637018 10014 raft_consensus.cc:1261] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240426 09:55:51.654060 10100 tablet_copy_service.cc:342] P d3a480e831d649139ff016fb68aee3ee: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-c75e7dd8bbe64a3eb2174cd4a218d51c received from {username='slave'} at 127.0.0.1:48202
I20240426 09:55:51.654505 10100 tablet_copy_service.cc:433] P d3a480e831d649139ff016fb68aee3ee: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-c75e7dd8bbe64a3eb2174cd4a218d51c on tablet c75e7dd8bbe64a3eb2174cd4a218d51c with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:51.660852 12594 consensus_queue.cc:1035] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 09:55:51.665051 9933 raft_consensus.cc:1261] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240426 09:55:51.670326 12544 consensus_queue.cc:1035] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:51.721992 12563 tablet_bootstrap.cc:492] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:51.723022 12563 tablet_bootstrap.cc:492] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:51.723834 12563 ts_tablet_manager.cc:1367] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.176s user 0.030s sys 0.003s
I20240426 09:55:51.727180 12563 raft_consensus.cc:348] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:51.728185 12563 raft_consensus.cc:729] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:51.739192 12563 consensus_queue.cc:260] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:51.744542 12563 ts_tablet_manager.cc:1398] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.020s user 0.002s sys 0.004s
I20240426 09:55:51.746074 9951 tablet_copy_service.cc:342] P f12521b6d51242629cf4297864df00b9: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-fb385fda44e94153ae3cd9ca21240939 received from {username='slave'} at 127.0.0.1:40928
I20240426 09:55:51.746457 9951 tablet_copy_service.cc:433] P f12521b6d51242629cf4297864df00b9: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-fb385fda44e94153ae3cd9ca21240939 on tablet fb385fda44e94153ae3cd9ca21240939 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:51.776139 10613 raft_consensus.cc:1203] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Deduplicated request from leader. Original: 7.4->[7.5-7.5] Dedup: 7.5->[]
I20240426 09:55:51.805032 10613 raft_consensus.cc:1203] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Deduplicated request from leader. Original: 7.5->[7.6-7.6] Dedup: 7.6->[]
I20240426 09:55:51.827241 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:52.076368 10014 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:52.076953 10014 raft_consensus.cc:595] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Received request to transfer leadership to TS d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:52.084731 9933 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
mode: GRACEFUL
new_leader_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:46580
I20240426 09:55:52.085291 9933 raft_consensus.cc:595] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Received request to transfer leadership to TS d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:52.111502 10082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:52.112105 10082 raft_consensus.cc:595] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:52.124408 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240426 09:55:52.126250 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:55:52.127061 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:52.165488 12544 raft_consensus.cc:979] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:52.165966 12544 raft_consensus.cc:1067] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:52.168119 10014 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:55:52.168633 10014 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:52.168939 10014 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:52.179155 10014 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:52.184897 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:52.185451 9937 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:52.186167 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:52.186707 10081 raft_consensus.cc:3032] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:55:52.187002 10081 raft_consensus.cc:729] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:52.187705 10081 consensus_queue.cc:260] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:52.188738 10081 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:52.185254 10014 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:52.214658 9937 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:55:52.216326 9982 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:52.217198 12479 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:52.217657 12479 raft_consensus.cc:686] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:52.218497 12479 consensus_queue.cc:237] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:52.220194 10081 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:55:52.229781 12585 raft_consensus.cc:1050] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:52.240522 12585 consensus_queue.cc:237] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 4, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:52.260371 10613 raft_consensus.cc:1261] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:55:52.262135 12354 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:52.271098 10081 raft_consensus.cc:1261] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:55:52.272280 12585 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 09:55:52.272682 10014 raft_consensus.cc:1261] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:55:52.275754 12585 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:52.291715 9839 catalog_manager.cc:5484] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 6 to 7, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 7 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:52.292801 12354 raft_consensus.cc:2932] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Committing config change with OpId 7.7: config changed from index 6 to 7, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:52.301441 10014 raft_consensus.cc:2932] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 6 to 7, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:52.311195 10089 raft_consensus.cc:2932] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 6 to 7, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:52.327569 10613 raft_consensus.cc:2932] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 6 to 7, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:52.341189 12577 raft_consensus.cc:1050] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:52.353245 9836 catalog_manager.cc:5484] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index 6 to 7, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:52.357218 12577 consensus_queue.cc:237] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 3, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:52.390420 10014 raft_consensus.cc:1261] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 09:55:52.391829 12594 consensus_queue.cc:1035] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:52.400585 10613 raft_consensus.cc:1261] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 09:55:52.409083 12594 consensus_queue.cc:1035] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:52.414940 9933 raft_consensus.cc:1261] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 09:55:52.416257 12594 consensus_queue.cc:1035] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:52.439126 12594 raft_consensus.cc:2932] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Committing config change with OpId 7.6: config changed from index 5 to 6, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:52.441107 10014 raft_consensus.cc:2932] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index 5 to 6, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:52.480827 12549 raft_consensus.cc:979] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9: : Instructing follower d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:52.480317 9839 catalog_manager.cc:5484] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index 5 to 6, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:52.481340 12549 raft_consensus.cc:1067] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Signalling peer d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:52.482801 10089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:59762
I20240426 09:55:52.483284 10089 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:52.483584 10089 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:52.488209 9937 raft_consensus.cc:2932] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index 5 to 6, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:52.489768 10089 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:52.495141 9935 consensus_queue.cc:237] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:52.500766 10009 raft_consensus.cc:1261] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 09:55:52.502751 10618 raft_consensus.cc:1261] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 09:55:52.505185 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:52.505759 10009 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:52.511605 10089 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:52.512104 10009 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:52.513181 10055 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:52.513505 12549 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 09:55:52.513923 12544 raft_consensus.cc:2781] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:52.514429 12544 raft_consensus.cc:686] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:52.515256 12544 consensus_queue.cc:237] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:52.519402 10613 raft_consensus.cc:2932] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index 5 to 6, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:52.521092 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:52.532847 9935 raft_consensus.cc:3032] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:52.533136 10081 consensus_queue.cc:237] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:52.534406 12354 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 09:55:52.545832 12585 raft_consensus.cc:2932] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Committing config change with OpId 7.8: config changed from index 7 to 8, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
W20240426 09:55:52.551801 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 49393.8us
I20240426 09:55:52.552583 10612 raft_consensus.cc:2932] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Committing config change with OpId 7.8: config changed from index 7 to 8, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:52.561661 9937 raft_consensus.cc:1261] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:55:52.562805 12594 consensus_queue.cc:1035] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:52.533213 9935 raft_consensus.cc:729] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:52.573262 9935 consensus_queue.cc:260] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:52.578251 10618 raft_consensus.cc:1261] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:55:52.579762 9935 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:52.592358 9935 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:55:52.605819 9839 catalog_manager.cc:5484] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 5 to 6, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 6 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
W20240426 09:55:52.609673 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 56404.5us
I20240426 09:55:52.619213 12594 consensus_queue.cc:1035] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:52.668764 10016 raft_consensus.cc:2932] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Committing config change with OpId 7.8: config changed from index 7 to 8, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:52.687758 9826 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fb385fda44e94153ae3cd9ca21240939 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:55:52.717164 9839 catalog_manager.cc:5484] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 7 to 8, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New cstate: current_term: 7 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:52.722900 12620 raft_consensus.cc:2932] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Committing config change with OpId 7.7: config changed from index 6 to 7, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:52.747633 10618 raft_consensus.cc:2932] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 6 to 7, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:52.770717 9835 catalog_manager.cc:5484] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 6 to 7, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New cstate: current_term: 7 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:52.773365 9823 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c75e7dd8bbe64a3eb2174cd4a218d51c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:55:52.774385 10077 tablet_service.cc:1514] Processing DeleteTablet for tablet fb385fda44e94153ae3cd9ca21240939 with delete_type TABLET_DATA_TOMBSTONED (TS d3a480e831d649139ff016fb68aee3ee not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:59730
I20240426 09:55:52.785039 9933 raft_consensus.cc:2932] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 6 to 7, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:52.787995 10089 raft_consensus.cc:1261] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 09:55:52.791882 12511 raft_consensus.cc:979] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:52.792325 12511 consensus_queue.cc:1035] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:52.792508 12617 raft_consensus.cc:1067] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Signalling peer d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:52.796239 12625 tablet_replica.cc:330] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee: stopping tablet replica
I20240426 09:55:52.797006 10082 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:59746
I20240426 09:55:52.797528 10082 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:52.797870 10082 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:52.804863 12625 raft_consensus.cc:2229] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Raft consensus shutting down.
I20240426 09:55:52.808714 12625 raft_consensus.cc:2258] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Raft consensus is shut down!
I20240426 09:55:52.808912 10082 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:52.819567 10082 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:52.827036 10006 tablet_service.cc:1514] Processing DeleteTablet for tablet c75e7dd8bbe64a3eb2174cd4a218d51c with delete_type TABLET_DATA_TOMBSTONED (TS 826f03b2fb9c4b9483623b041a1bf80e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34362
I20240426 09:55:52.831503 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:52.832082 9935 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:52.832556 12625 ts_tablet_manager.cc:1875] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:55:52.837882 9935 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:55:52.838081 9933 raft_consensus.cc:1261] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 09:55:52.841105 10056 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:52.841540 12627 tablet_replica.cc:330] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e: stopping tablet replica
I20240426 09:55:52.841804 12624 raft_consensus.cc:2781] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:52.842307 12624 raft_consensus.cc:686] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:52.842583 12627 raft_consensus.cc:2229] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Raft consensus shutting down.
I20240426 09:55:52.843390 12627 raft_consensus.cc:2258] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Raft consensus is shut down!
I20240426 09:55:52.843132 12624 consensus_queue.cc:237] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:52.843930 12617 consensus_queue.cc:1035] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:52.846416 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:52.850345 10015 raft_consensus.cc:3032] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:55:52.850638 10015 raft_consensus.cc:729] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:52.851300 10015 consensus_queue.cc:260] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:52.852428 10015 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:52.859153 10015 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:55:52.868234 12627 ts_tablet_manager.cc:1875] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:55:52.882552 9838 catalog_manager.cc:5484] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 6 to 7, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 7 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:52.943817 12625 ts_tablet_manager.cc:1888] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.7
I20240426 09:55:52.948108 12625 log.cc:1196] T fb385fda44e94153ae3cd9ca21240939 P d3a480e831d649139ff016fb68aee3ee: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-2-root/wals/fb385fda44e94153ae3cd9ca21240939
I20240426 09:55:52.949841 9823 catalog_manager.cc:4830] TS d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): tablet fb385fda44e94153ae3cd9ca21240939 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:55:52.947288 12627 ts_tablet_manager.cc:1888] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.6
I20240426 09:55:52.950701 12627 log.cc:1196] T c75e7dd8bbe64a3eb2174cd4a218d51c P 826f03b2fb9c4b9483623b041a1bf80e: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-1-root/wals/c75e7dd8bbe64a3eb2174cd4a218d51c
I20240426 09:55:52.957108 9825 catalog_manager.cc:4830] TS 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): tablet c75e7dd8bbe64a3eb2174cd4a218d51c (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:55:53.123668 10015 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240426 09:55:53.125097 12624 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:53.132558 9937 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240426 09:55:53.134133 12624 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:53.766894 9941 raft_consensus.cc:1261] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 09:55:53.605417 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:53.819193 12621 consensus_queue.cc:1035] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240426 09:55:53.879081 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 74862.3us
W20240426 09:55:53.969715 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 48838us
W20240426 09:55:54.005409 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 60443.8us
I20240426 09:55:54.031498 10008 raft_consensus.cc:1261] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 09:55:54.036002 10015 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:54.036550 10015 raft_consensus.cc:595] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:54.052398 10082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:54.053843 10082 raft_consensus.cc:595] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
W20240426 09:55:54.054956 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 64960us
I20240426 09:55:54.082248 12644 raft_consensus.cc:979] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:54.082701 12644 raft_consensus.cc:1067] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:54.086860 12621 consensus_queue.cc:1035] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:54.088689 9940 consensus_queue.cc:237] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:54.101022 10084 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:54.101558 10084 raft_consensus.cc:595] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:54.103873 10008 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:55:54.104388 10008 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:54.104704 10008 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:54.108312 10014 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:54.108901 10014 raft_consensus.cc:595] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:54.110049 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 09:55:54.111730 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:55:54.112329 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:54.112468 10008 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:54.114719 10008 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:54.114913 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:54.115818 9941 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:54.121448 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:54.122035 10082 raft_consensus.cc:3032] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:55:54.122334 10082 raft_consensus.cc:729] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:54.125237 9941 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:55:54.126209 10082 consensus_queue.cc:260] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:54.127593 10082 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
W20240426 09:55:54.128054 9905 consensus_peers.cc:485] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf8634deb0a5425db9c47915d49386d5. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:54.131088 10084 raft_consensus.cc:1261] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 09:55:54.132444 12635 consensus_queue.cc:1035] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:54.134438 10008 raft_consensus.cc:1261] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 09:55:54.134743 9982 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:54.139272 12511 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:54.139761 12511 raft_consensus.cc:686] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:54.140321 12635 consensus_queue.cc:1035] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:54.140558 12511 consensus_queue.cc:237] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:54.157975 10082 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:55:54.169768 12585 raft_consensus.cc:2932] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:54.172456 10084 raft_consensus.cc:2932] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:54.184542 10015 raft_consensus.cc:2932] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
W20240426 09:55:54.189671 9887 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
I20240426 09:55:54.193552 9838 catalog_manager.cc:5484] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 reported cstate change: config changed from index -1 to 4, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 5 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240426 09:55:54.198122 9887 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 69700f6b120b4f4ebc0b43b14166f4c9, TS f12521b6d51242629cf4297864df00b9 -> TS 1fee3da01ebc463396397057fdb81a68 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:54.198454 9887 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 69700f6b120b4f4ebc0b43b14166f4c9, TS f12521b6d51242629cf4297864df00b9 -> TS 1fee3da01ebc463396397057fdb81a68 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:54.227605 9887 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet bc3e15fcf67145f38946b62f0a5e9149, TS d3a480e831d649139ff016fb68aee3ee -> TS 1fee3da01ebc463396397057fdb81a68 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:54.227916 9887 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet bc3e15fcf67145f38946b62f0a5e9149, TS d3a480e831d649139ff016fb68aee3ee -> TS 1fee3da01ebc463396397057fdb81a68 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:54.242430 9839 catalog_manager.cc:5484] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 6 to 7, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 7 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
W20240426 09:55:54.248940 9887 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 5e80ebfd190643a99ef02b6ea2876426, TS f12521b6d51242629cf4297864df00b9 -> TS 1fee3da01ebc463396397057fdb81a68 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:55:54.249265 9887 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 5e80ebfd190643a99ef02b6ea2876426, TS f12521b6d51242629cf4297864df00b9 -> TS 1fee3da01ebc463396397057fdb81a68 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:54.253083 12511 raft_consensus.cc:979] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:54.253561 12511 raft_consensus.cc:1067] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Signalling peer f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:54.255414 9940 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:47242
I20240426 09:55:54.255911 9940 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:54.256242 9940 raft_consensus.cc:3037] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:54.264333 9940 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:54.268585 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:54.269104 10612 raft_consensus.cc:3037] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:54.269690 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:54.270228 10015 raft_consensus.cc:3032] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:55:54.270507 10015 raft_consensus.cc:729] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:54.271142 10015 consensus_queue.cc:260] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:54.272161 10015 raft_consensus.cc:3037] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:55:54.276758 10612 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:54.279075 9905 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:54.280050 12585 raft_consensus.cc:2781] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:55:54.280514 12585 raft_consensus.cc:686] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:54.269055 9940 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:54.309137 12643 raft_consensus.cc:979] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:54.309654 12620 raft_consensus.cc:1067] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:54.281276 12585 consensus_queue.cc:237] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:54.323233 10008 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:55:54.323751 10008 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:54.324100 10008 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:54.356710 10015 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:55:54.387300 9835 catalog_manager.cc:5484] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 3 to 4, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 4 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:54.386940 10008 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:54.398801 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:54.398977 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:54.399403 10084 raft_consensus.cc:3032] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:55:54.399777 10084 raft_consensus.cc:729] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:54.400480 10084 consensus_queue.cc:260] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:54.401522 10084 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:54.411106 10084 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:55:54.413827 9979 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:54.414528 12657 raft_consensus.cc:2781] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:54.415009 12657 raft_consensus.cc:686] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:54.415831 12657 consensus_queue.cc:237] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:54.421483 10008 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:54.399636 9936 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:54.452421 9936 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:55:54.456001 9835 catalog_manager.cc:5484] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 6 to 7, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 7 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
W20240426 09:55:54.538282 10037 debug-util.cc:398] Leaking SignalData structure 0x7b08003014c0 after lost signal to thread 8761
W20240426 09:55:54.539412 10037 debug-util.cc:398] Leaking SignalData structure 0x7b08002f34c0 after lost signal to thread 9873
W20240426 09:55:54.540716 10037 debug-util.cc:398] Leaking SignalData structure 0x7b0800169820 after lost signal to thread 9965
W20240426 09:55:54.541790 10037 debug-util.cc:398] Leaking SignalData structure 0x7b0800162fc0 after lost signal to thread 10040
W20240426 09:55:54.542883 10037 debug-util.cc:398] Leaking SignalData structure 0x7b080037d9c0 after lost signal to thread 10114
W20240426 09:55:54.544016 10037 debug-util.cc:398] Leaking SignalData structure 0x7b0800331500 after lost signal to thread 10644
I20240426 09:55:54.592793 12511 raft_consensus.cc:979] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:54.593300 12511 raft_consensus.cc:1067] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Signalling peer f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:54.595769 9936 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:47242
I20240426 09:55:54.596271 9936 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:54.596585 9936 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:54.631165 9936 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:54.633392 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:54.633961 10008 raft_consensus.cc:3032] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:54.634260 10008 raft_consensus.cc:729] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:54.634941 10008 consensus_queue.cc:260] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:54.635982 10008 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:54.640379 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:54.640954 10089 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:54.641609 10084 raft_consensus.cc:1261] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:55:54.643050 12657 consensus_queue.cc:1035] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:54.647138 10089 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:55:54.648267 9904 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:54.648995 12636 raft_consensus.cc:2781] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:54.650110 12636 raft_consensus.cc:686] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 6 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:54.651860 10008 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:55:54.652065 12636 consensus_queue.cc:237] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:54.639534 9936 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:54.699491 12674 ts_tablet_manager.cc:902] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113)
I20240426 09:55:54.747010 9839 catalog_manager.cc:5484] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 5 to 6, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 6 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:54.753041 9941 raft_consensus.cc:1261] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:55:54.757221 12511 consensus_queue.cc:1035] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:54.799170 10612 raft_consensus.cc:1261] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 09:55:54.800554 12636 consensus_queue.cc:1035] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:54.831008 9936 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 09:55:54.832450 12640 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:54.896535 10008 raft_consensus.cc:1261] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 09:55:54.906447 12585 consensus_queue.cc:1035] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:54.937348 10082 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 09:55:54.942734 12640 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20240426 09:55:55.048435 12674 tablet_copy_client.cc:308] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.129:38113
W20240426 09:55:55.050326 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 108852us
W20240426 09:55:55.050720 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 93590.1us
I20240426 09:55:55.055351 9951 tablet_copy_service.cc:140] P f12521b6d51242629cf4297864df00b9: Received BeginTabletCopySession request for tablet cf8634deb0a5425db9c47915d49386d5 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:40928)
I20240426 09:55:55.055867 9951 tablet_copy_service.cc:161] P f12521b6d51242629cf4297864df00b9: Beginning new tablet copy session on tablet cf8634deb0a5425db9c47915d49386d5 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:40928: session id = 1fee3da01ebc463396397057fdb81a68-cf8634deb0a5425db9c47915d49386d5
I20240426 09:55:55.063694 9951 tablet_copy_source_session.cc:205] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:55.071069 10014 raft_consensus.cc:1261] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:55:55.075273 12674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf8634deb0a5425db9c47915d49386d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:55.104880 12585 consensus_queue.cc:1035] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:55.123494 12674 tablet_copy_client.cc:783] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:55.129551 12674 tablet_copy_client.cc:647] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:55.138418 12674 tablet_copy_client.cc:523] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:55.155018 12674 tablet_bootstrap.cc:492] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:55.208539 10084 raft_consensus.cc:1261] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:55:55.216415 12637 consensus_queue.cc:1035] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:55.220150 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:55.285308 12674 tablet_bootstrap.cc:492] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:55.286360 12674 tablet_bootstrap.cc:492] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:55.287139 12674 ts_tablet_manager.cc:1367] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.132s user 0.032s sys 0.014s
I20240426 09:55:55.289989 12674 raft_consensus.cc:348] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:55.302639 12674 raft_consensus.cc:729] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:55.303459 12674 consensus_queue.cc:260] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:55.306418 12674 ts_tablet_manager.cc:1398] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.019s user 0.004s sys 0.001s
I20240426 09:55:55.308101 9951 tablet_copy_service.cc:342] P f12521b6d51242629cf4297864df00b9: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-cf8634deb0a5425db9c47915d49386d5 received from {username='slave'} at 127.0.0.1:40928
I20240426 09:55:55.308549 9951 tablet_copy_service.cc:433] P f12521b6d51242629cf4297864df00b9: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-cf8634deb0a5425db9c47915d49386d5 on tablet cf8634deb0a5425db9c47915d49386d5 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:55.395128 9940 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
mode: GRACEFUL
new_leader_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:46580
I20240426 09:55:55.396816 9940 raft_consensus.cc:595] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Received request to transfer leadership to TS d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:55.422619 9936 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:46580
I20240426 09:55:55.423213 9936 raft_consensus.cc:595] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 6 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:55.428385 10014 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:55.428898 10014 raft_consensus.cc:595] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Received request to transfer leadership to TS d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:55.438385 10014 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:55.438935 10014 raft_consensus.cc:595] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Received request to transfer leadership to TS d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:55.441805 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 09:55:55.443392 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:55:55.447268 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:55.461308 12701 raft_consensus.cc:979] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9: : Instructing follower d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:55.461762 12701 raft_consensus.cc:1067] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Signalling peer d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:55.463222 10082 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:59762
I20240426 09:55:55.463733 10082 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:55.464051 10082 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:55.480051 10082 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:55.484489 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:55.485034 9939 raft_consensus.cc:3032] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:55:55.485291 9939 raft_consensus.cc:729] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:55.486028 9939 consensus_queue.cc:260] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:55.487236 9939 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:55.493261 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: true dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:55.493852 10612 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:55.519987 10612 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:55:55.520756 10082 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:55.522666 10054 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:55.543516 12698 raft_consensus.cc:2781] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:55:55.547386 12698 raft_consensus.cc:686] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:55.548244 12698 consensus_queue.cc:237] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:55.558259 12637 raft_consensus.cc:979] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:55.558725 12637 raft_consensus.cc:1067] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 6 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:55.560432 10008 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34390
I20240426 09:55:55.560925 10008 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:55.561230 10008 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:55.568449 10008 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:55.570907 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:55.571043 10008 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:55.571475 9940 raft_consensus.cc:3032] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:55:55.571797 9940 raft_consensus.cc:729] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:55.572443 9940 consensus_queue.cc:260] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:55.573292 9940 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:55.577237 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:55.577783 10084 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:55.581018 10612 raft_consensus.cc:1203] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 5 LEARNER]: Deduplicated request from leader. Original: 5.3->[5.4-5.4] Dedup: 5.4->[]
I20240426 09:55:55.588833 9940 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:55:55.591035 9982 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:55.592025 12690 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:55.592456 10084 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:55:55.592470 12690 raft_consensus.cc:686] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:55.595971 12690 consensus_queue.cc:237] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:55.609745 12640 raft_consensus.cc:979] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:55.610211 12640 raft_consensus.cc:1067] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Signalling peer d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:55.613375 9939 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:55:55.637552 9839 catalog_manager.cc:5484] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 6 to 7, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 7 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:55.640878 10082 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:59746
I20240426 09:55:55.641695 10082 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:55.642102 10082 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:55.626521 9838 catalog_manager.cc:5484] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 7 to 8, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 8 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:55.661710 10082 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:55.667347 10082 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:55.679884 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:55.680505 10008 raft_consensus.cc:3032] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:55:55.680778 10008 raft_consensus.cc:729] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:55.681427 10008 consensus_queue.cc:260] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:55.683100 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:55.683670 9940 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:55.682582 10008 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:55.698238 12642 raft_consensus.cc:979] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:55.698745 12642 raft_consensus.cc:1067] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Signalling peer d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:55.700723 10089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:59746
I20240426 09:55:55.701249 10089 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:55.701557 10089 raft_consensus.cc:3037] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:55.706354 10089 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:55.711264 9940 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:55:55.719203 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:55.720120 9935 raft_consensus.cc:3037] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:55.719435 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:55.723124 10014 raft_consensus.cc:3032] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:55:55.723423 10014 raft_consensus.cc:729] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:55.724103 10014 consensus_queue.cc:260] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:55.725121 10014 raft_consensus.cc:3037] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:55:55.732218 9935 raft_consensus.cc:2454] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:55.733506 10056 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:55.735391 12698 raft_consensus.cc:2781] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:55:55.735857 12698 raft_consensus.cc:686] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:55.736610 12698 consensus_queue.cc:237] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:55.738901 10056 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:55.740494 12624 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:55:55.740962 12624 raft_consensus.cc:686] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:55.741703 12624 consensus_queue.cc:237] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:55.718628 10089 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:55.757740 10014 raft_consensus.cc:2454] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:55:55.758462 10008 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:55:55.794426 9838 catalog_manager.cc:5484] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 4 to 5, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 5 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:55.796041 9838 catalog_manager.cc:5484] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 7 to 8, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 8 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:56.023664 10612 raft_consensus.cc:1261] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 09:55:56.026599 12698 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 09:55:56.073937 9935 raft_consensus.cc:1261] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 09:55:56.081331 12698 consensus_queue.cc:1035] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 09:55:56.119352 10089 raft_consensus.cc:1261] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 09:55:56.121770 12690 consensus_queue.cc:1035] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:56.132156 9935 raft_consensus.cc:1261] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 09:55:56.135176 12695 consensus_queue.cc:1035] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:56.169955 12636 raft_consensus.cc:1050] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:56.172412 12636 consensus_queue.cc:237] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 2, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:56.196497 9935 raft_consensus.cc:1261] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 09:55:56.209070 12696 consensus_queue.cc:1035] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:56.218931 10612 raft_consensus.cc:1261] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 5 LEARNER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240426 09:55:56.220258 10014 raft_consensus.cc:1261] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240426 09:55:56.220479 12637 consensus_queue.cc:1035] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:56.229219 10089 raft_consensus.cc:1261] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240426 09:55:56.232880 12636 consensus_queue.cc:1035] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:56.279568 12701 consensus_queue.cc:1035] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:56.288450 10008 raft_consensus.cc:1261] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 09:55:56.289928 12624 consensus_queue.cc:1035] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:56.296940 9939 raft_consensus.cc:1261] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 09:55:56.332841 12624 consensus_queue.cc:1035] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:56.344591 10015 raft_consensus.cc:1261] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 09:55:56.347180 12704 consensus_queue.cc:1035] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:56.396234 12636 raft_consensus.cc:2932] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Committing config change with OpId 5.5: config changed from index 4 to 5, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:56.403357 10013 raft_consensus.cc:2932] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index 4 to 5, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:56.411453 10084 raft_consensus.cc:2932] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index 4 to 5, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:56.427644 10612 raft_consensus.cc:2932] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index 4 to 5, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:56.472257 9838 catalog_manager.cc:5484] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index 4 to 5, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } attrs { replace: true } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:56.572428 9936 consensus_queue.cc:237] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:56.582163 10618 raft_consensus.cc:1261] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 09:55:56.584311 12725 consensus_queue.cc:1035] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:56.586621 10084 raft_consensus.cc:1261] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 09:55:56.594234 12725 consensus_queue.cc:1035] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:56.606402 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:56.613075 12701 raft_consensus.cc:2932] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 5 to 6, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:56.627681 10082 raft_consensus.cc:2932] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:56.644446 10618 raft_consensus.cc:2932] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:56.658071 9826 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf8634deb0a5425db9c47915d49386d5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:55:56.665760 9836 catalog_manager.cc:5484] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index 5 to 6, VOTER 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) evicted. New cstate: current_term: 5 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:55:56.676884 10006 tablet_service.cc:1514] Processing DeleteTablet for tablet cf8634deb0a5425db9c47915d49386d5 with delete_type TABLET_DATA_TOMBSTONED (TS 826f03b2fb9c4b9483623b041a1bf80e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34362
I20240426 09:55:56.685112 12735 tablet_replica.cc:330] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e: stopping tablet replica
I20240426 09:55:56.690511 12735 raft_consensus.cc:2229] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Raft consensus shutting down.
I20240426 09:55:56.695333 12735 raft_consensus.cc:2258] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Raft consensus is shut down!
I20240426 09:55:56.706333 12735 ts_tablet_manager.cc:1875] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:55:56.725749 12735 ts_tablet_manager.cc:1888] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.5
I20240426 09:55:56.727070 12735 log.cc:1196] T cf8634deb0a5425db9c47915d49386d5 P 826f03b2fb9c4b9483623b041a1bf80e: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-1-root/wals/cf8634deb0a5425db9c47915d49386d5
I20240426 09:55:56.729295 9825 catalog_manager.cc:4830] TS 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411): tablet cf8634deb0a5425db9c47915d49386d5 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:55:56.790810 10082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:56.791394 10082 raft_consensus.cc:595] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:56.811951 10082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:56.814218 10082 raft_consensus.cc:595] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:56.823297 10008 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:56.823858 10008 raft_consensus.cc:595] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:56.825326 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240426 09:55:56.826951 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:55:56.827549 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:57.003908 12704 raft_consensus.cc:979] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:57.004472 12698 raft_consensus.cc:1067] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:57.019532 10014 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:55:57.020080 10014 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:57.020411 10014 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:57.026821 10014 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:57.031476 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:57.031901 10014 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:57.032467 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:57.032042 9935 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:57.034755 10084 raft_consensus.cc:3032] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:55:57.035317 10084 raft_consensus.cc:729] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:57.036054 10084 consensus_queue.cc:260] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:57.037067 10084 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:57.047971 9935 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:55:57.049188 9982 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:57.049937 12695 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:55:57.050391 12695 raft_consensus.cc:686] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:57.051182 12695 consensus_queue.cc:237] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:57.048158 10084 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:55:57.113459 9839 catalog_manager.cc:5484] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 7 to 8, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 8 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:57.221001 12698 raft_consensus.cc:979] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee: : Instructing follower f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:57.221447 12698 raft_consensus.cc:1067] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Signalling peer f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:57.223117 9934 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:47248
I20240426 09:55:57.223661 9934 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:57.223994 9934 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:57.230536 9934 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:57.232580 9934 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:57.232810 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:57.242125 10014 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:57.233428 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:57.242650 10084 raft_consensus.cc:3032] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:55:57.242975 10084 raft_consensus.cc:729] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:57.243633 10084 consensus_queue.cc:260] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:57.244647 10084 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:57.248385 10014 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 8.
I20240426 09:55:57.249725 9906 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:57.250675 12726 raft_consensus.cc:2781] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:55:57.251169 12726 raft_consensus.cc:686] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [term 8 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:57.251919 12726 consensus_queue.cc:237] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:57.261960 10084 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 8.
I20240426 09:55:57.289208 9839 catalog_manager.cc:5484] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 7 to 8, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 8 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:57.303035 12695 raft_consensus.cc:979] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:57.305150 12695 raft_consensus.cc:1067] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Signalling peer f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:57.306665 9934 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:47242
I20240426 09:55:57.307188 9934 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:57.307495 9934 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:57.314283 9934 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:57.317607 9934 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:57.319366 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:57.319978 10014 raft_consensus.cc:3032] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:55:57.320276 10014 raft_consensus.cc:729] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:57.321146 10014 consensus_queue.cc:260] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:57.323765 10014 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:57.324440 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:57.325206 10084 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:55:57.345312 10084 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:55:57.345767 10014 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:55:57.346650 9904 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:57.357110 12726 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:55:57.357597 12726 raft_consensus.cc:686] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:57.358460 12726 consensus_queue.cc:237] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:57.397256 9839 catalog_manager.cc:5484] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 6 to 7, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 7 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:57.500762 10084 raft_consensus.cc:1261] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 09:55:57.502983 12695 consensus_queue.cc:1035] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:57.549046 9934 raft_consensus.cc:1261] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 09:55:57.550422 12642 consensus_queue.cc:1035] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:57.648090 10014 raft_consensus.cc:1261] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240426 09:55:57.649453 12739 consensus_queue.cc:1035] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:57.689533 10084 raft_consensus.cc:1261] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240426 09:55:57.693212 12726 consensus_queue.cc:1035] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:57.850746 10014 consensus_queue.cc:237] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:57.867846 10084 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20240426 09:55:57.868356 9980 consensus_peers.cc:485] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e80ebfd190643a99ef02b6ea2876426. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:57.869460 12745 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:57.873950 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:57.889209 9934 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240426 09:55:57.890939 12695 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:57.912703 10084 raft_consensus.cc:2932] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Committing config change with OpId 8.6: config changed from index -1 to 6, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:57.921510 12695 raft_consensus.cc:2932] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Committing config change with OpId 8.6: config changed from index -1 to 6, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:57.929615 10008 raft_consensus.cc:1261] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 09:55:57.930918 12739 consensus_queue.cc:1035] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:57.940745 9839 catalog_manager.cc:5484] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index -1 to 6, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 8 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:57.973971 10082 raft_consensus.cc:1261] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 09:55:57.976698 12726 consensus_queue.cc:1035] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:57.984669 10013 consensus_queue.cc:237] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
W20240426 09:55:58.045931 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 53707.1us
W20240426 09:55:58.062361 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 73793.4us
I20240426 09:55:58.074389 9936 raft_consensus.cc:2932] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Committing config change with OpId 8.6: config changed from index -1 to 6, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:58.103860 9935 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 09:55:58.110639 10081 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 09:55:58.115948 12745 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240426 09:55:58.118055 9980 consensus_peers.cc:485] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3dc043e5f7b241afb7119ba8b06ed37f. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:58.121770 12755 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20240426 09:55:58.133754 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 69029.9us
I20240426 09:55:58.195222 10008 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:58.195824 10008 raft_consensus.cc:595] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Received request to transfer leadership to TS d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:58.196733 12690 raft_consensus.cc:2932] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Committing config change with OpId 8.8: config changed from index -1 to 8, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:58.208927 9936 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
mode: GRACEFUL
new_leader_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:46580
I20240426 09:55:58.209632 9936 raft_consensus.cc:595] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Received request to transfer leadership to TS d3a480e831d649139ff016fb68aee3ee
I20240426 09:55:58.215487 10084 raft_consensus.cc:2932] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Committing config change with OpId 8.8: config changed from index -1 to 8, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:58.226989 10089 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:58.227589 10089 raft_consensus.cc:595] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:58.227468 10008 consensus_queue.cc:237] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:58.229686 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240426 09:55:58.232719 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:55:58.231875 9839 catalog_manager.cc:5484] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index -1 to 8, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 8 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:58.233130 9934 raft_consensus.cc:2932] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Committing config change with OpId 8.8: config changed from index -1 to 8, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:58.239826 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240426 09:55:58.241981 9980 consensus_peers.cc:485] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69700f6b120b4f4ebc0b43b14166f4c9. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:58.242980 9936 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:55:58.244483 12690 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:58.249945 10084 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:55:58.251195 12690 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:58.286532 12690 raft_consensus.cc:2932] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Committing config change with OpId 7.7: config changed from index -1 to 7, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:58.300065 10089 raft_consensus.cc:2932] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index -1 to 7, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:58.316463 10008 consensus_queue.cc:237] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:58.331199 9836 catalog_manager.cc:5484] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee reported cstate change: config changed from index -1 to 7, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 7 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:58.358379 9934 raft_consensus.cc:2932] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index -1 to 7, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
W20240426 09:55:58.365995 9980 consensus_peers.cc:485] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8574d769068f477e859574c959d5880e. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:55:58.369167 9936 raft_consensus.cc:1261] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240426 09:55:58.369871 10084 raft_consensus.cc:1261] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240426 09:55:58.371665 12755 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:58.373131 12770 ts_tablet_manager.cc:902] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:58.373411 12755 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:55:58.384474 12704 raft_consensus.cc:979] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:58.384964 12704 raft_consensus.cc:1067] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:58.387290 10008 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:55:58.387861 10008 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:58.388236 10008 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:58.395332 10008 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:58.394726 12755 raft_consensus.cc:2932] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:58.397794 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:58.398319 9935 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:58.398545 10008 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:58.398710 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:58.399263 10089 raft_consensus.cc:3032] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:55:58.399585 10089 raft_consensus.cc:729] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:58.400244 10089 consensus_queue.cc:260] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:58.401368 10089 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:55:58.404367 9935 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:58.405588 9982 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:55:58.406797 12690 raft_consensus.cc:2781] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:55:58.407047 10089 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:55:58.397622 9936 raft_consensus.cc:2932] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:58.412084 12690 raft_consensus.cc:686] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:58.412994 12690 consensus_queue.cc:237] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:58.429286 9839 catalog_manager.cc:5484] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 reported cstate change: config changed from index -1 to 5, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New cstate: current_term: 6 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:58.433866 10081 raft_consensus.cc:2932] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:55:58.471151 12770 tablet_copy_client.cc:308] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.130:44411
I20240426 09:55:58.473107 10026 tablet_copy_service.cc:140] P 826f03b2fb9c4b9483623b041a1bf80e: Received BeginTabletCopySession request for tablet 5e80ebfd190643a99ef02b6ea2876426 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:36762)
I20240426 09:55:58.473630 10026 tablet_copy_service.cc:161] P 826f03b2fb9c4b9483623b041a1bf80e: Beginning new tablet copy session on tablet 5e80ebfd190643a99ef02b6ea2876426 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:36762: session id = 1fee3da01ebc463396397057fdb81a68-5e80ebfd190643a99ef02b6ea2876426
I20240426 09:55:58.474939 9836 catalog_manager.cc:5484] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 5 to 6, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 6 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:58.486809 10026 tablet_copy_source_session.cc:205] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:58.516144 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e80ebfd190643a99ef02b6ea2876426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:58.563329 12778 ts_tablet_manager.cc:902] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:58.557104 12770 tablet_copy_client.cc:783] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:58.565021 12770 tablet_copy_client.cc:647] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:58.572067 12770 tablet_copy_client.cc:523] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:58.611330 12770 tablet_bootstrap.cc:492] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:58.616317 12778 tablet_copy_client.cc:308] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.130:44411
I20240426 09:55:58.617879 10026 tablet_copy_service.cc:140] P 826f03b2fb9c4b9483623b041a1bf80e: Received BeginTabletCopySession request for tablet 3dc043e5f7b241afb7119ba8b06ed37f from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:36762)
I20240426 09:55:58.618434 10026 tablet_copy_service.cc:161] P 826f03b2fb9c4b9483623b041a1bf80e: Beginning new tablet copy session on tablet 3dc043e5f7b241afb7119ba8b06ed37f from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:36762: session id = 1fee3da01ebc463396397057fdb81a68-3dc043e5f7b241afb7119ba8b06ed37f
I20240426 09:55:58.625306 12725 raft_consensus.cc:979] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9: : Instructing follower d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:58.627009 12725 raft_consensus.cc:1067] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Signalling peer d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:58.628543 10081 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:59762
I20240426 09:55:58.629034 10081 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:58.629351 10081 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:58.636891 10026 tablet_copy_source_session.cc:205] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:58.643255 10081 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:58.645596 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:58.646131 9934 raft_consensus.cc:3032] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:55:58.646291 10081 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:58.646478 9934 raft_consensus.cc:729] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:58.647339 9934 consensus_queue.cc:260] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:58.648337 9934 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:58.651584 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:58.652130 10008 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:58.658321 9934 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:55:58.667507 12778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dc043e5f7b241afb7119ba8b06ed37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:58.667526 10008 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:55:58.671761 12642 raft_consensus.cc:979] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:58.672246 12755 raft_consensus.cc:1067] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Signalling peer d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:55:58.673537 10055 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:58.675380 12757 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:55:58.675832 12757 raft_consensus.cc:686] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:58.676570 12757 consensus_queue.cc:237] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:58.681250 10089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:59746
I20240426 09:55:58.681783 10089 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:58.682114 10089 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:55:58.692890 9836 catalog_manager.cc:5484] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 7 to 8, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 8 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:55:58.699167 12778 tablet_copy_client.cc:783] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:58.703013 12778 tablet_copy_client.cc:647] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:58.719697 12778 tablet_copy_client.cc:523] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:58.722827 10089 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:58.726828 10089 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:58.727691 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:55:58.728289 10007 raft_consensus.cc:3032] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Stepping down as leader of term 8
I20240426 09:55:58.728543 10007 raft_consensus.cc:729] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:58.729223 10007 consensus_queue.cc:260] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:58.730243 10007 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:55:58.734238 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:58.734946 9935 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:55:58.755689 10007 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 9.
I20240426 09:55:58.757469 12784 ts_tablet_manager.cc:902] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:58.759341 12778 tablet_bootstrap.cc:492] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:58.761513 9935 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 9.
I20240426 09:55:58.771687 12770 tablet_bootstrap.cc:492] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=1 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:58.784927 12770 tablet_bootstrap.cc:492] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:58.786595 10055 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:58.787111 12770 ts_tablet_manager.cc:1367] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.175s user 0.028s sys 0.008s
I20240426 09:55:58.788257 12757 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader election won for term 9
I20240426 09:55:58.790005 12757 raft_consensus.cc:686] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:58.790910 12757 consensus_queue.cc:237] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:58.811218 9935 raft_consensus.cc:1261] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:55:58.812858 12690 consensus_queue.cc:1035] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:58.790256 12770 raft_consensus.cc:348] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 8 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:58.821005 12770 raft_consensus.cc:729] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 8 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:58.820564 12784 tablet_copy_client.cc:308] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.130:44411
I20240426 09:55:58.822607 10026 tablet_copy_service.cc:140] P 826f03b2fb9c4b9483623b041a1bf80e: Received BeginTabletCopySession request for tablet 69700f6b120b4f4ebc0b43b14166f4c9 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:36762)
I20240426 09:55:58.823158 10026 tablet_copy_service.cc:161] P 826f03b2fb9c4b9483623b041a1bf80e: Beginning new tablet copy session on tablet 69700f6b120b4f4ebc0b43b14166f4c9 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:36762: session id = 1fee3da01ebc463396397057fdb81a68-69700f6b120b4f4ebc0b43b14166f4c9
I20240426 09:55:58.832103 10026 tablet_copy_source_session.cc:205] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:58.843323 12770 consensus_queue.cc:260] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:58.845806 12770 ts_tablet_manager.cc:1398] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.058s user 0.004s sys 0.001s
I20240426 09:55:58.851161 12784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69700f6b120b4f4ebc0b43b14166f4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:55:58.861713 10026 tablet_copy_service.cc:342] P 826f03b2fb9c4b9483623b041a1bf80e: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-5e80ebfd190643a99ef02b6ea2876426 received from {username='slave'} at 127.0.0.1:36762
I20240426 09:55:58.862171 10026 tablet_copy_service.cc:433] P 826f03b2fb9c4b9483623b041a1bf80e: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-5e80ebfd190643a99ef02b6ea2876426 on tablet 5e80ebfd190643a99ef02b6ea2876426 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:58.877701 12770 ts_tablet_manager.cc:902] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:55:58.878906 10081 raft_consensus.cc:1261] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:55:58.880299 12642 consensus_queue.cc:1035] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:55:58.892834 9836 catalog_manager.cc:5484] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 8 to 9, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 9 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
W20240426 09:55:59.019109 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 104959us
I20240426 09:55:59.033069 12784 tablet_copy_client.cc:783] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:59.039482 12784 tablet_copy_client.cc:647] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:59.047331 12778 tablet_bootstrap.cc:492] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:59.048204 12778 tablet_bootstrap.cc:492] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:59.048913 12778 ts_tablet_manager.cc:1367] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.290s user 0.040s sys 0.019s
I20240426 09:55:59.053755 12784 tablet_copy_client.cc:523] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:59.094749 12778 raft_consensus.cc:348] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 8 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:59.095716 12778 raft_consensus.cc:729] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 8 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:59.096333 12778 consensus_queue.cc:260] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:59.130681 10013 raft_consensus.cc:1261] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240426 09:55:59.132841 12704 consensus_queue.cc:1035] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:59.145193 12778 ts_tablet_manager.cc:1398] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.096s user 0.007s sys 0.000s
W20240426 09:55:59.154490 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 48420.9us
I20240426 09:55:59.150192 10026 tablet_copy_service.cc:342] P 826f03b2fb9c4b9483623b041a1bf80e: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-3dc043e5f7b241afb7119ba8b06ed37f received from {username='slave'} at 127.0.0.1:36762
I20240426 09:55:59.161943 10026 tablet_copy_service.cc:433] P 826f03b2fb9c4b9483623b041a1bf80e: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-3dc043e5f7b241afb7119ba8b06ed37f on tablet 3dc043e5f7b241afb7119ba8b06ed37f with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:59.158226 12770 tablet_copy_client.cc:308] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.130:44411
I20240426 09:55:59.169291 10026 tablet_copy_service.cc:140] P 826f03b2fb9c4b9483623b041a1bf80e: Received BeginTabletCopySession request for tablet 8574d769068f477e859574c959d5880e from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:36762)
I20240426 09:55:59.171368 10026 tablet_copy_service.cc:161] P 826f03b2fb9c4b9483623b041a1bf80e: Beginning new tablet copy session on tablet 8574d769068f477e859574c959d5880e from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:36762: session id = 1fee3da01ebc463396397057fdb81a68-8574d769068f477e859574c959d5880e
W20240426 09:55:59.177521 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 71201.9us
I20240426 09:55:59.209434 9936 raft_consensus.cc:1261] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240426 09:55:59.212193 12757 consensus_queue.cc:1035] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:55:59.229516 10026 tablet_copy_source_session.cc:205] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:55:59.235985 12784 tablet_bootstrap.cc:492] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
W20240426 09:55:59.268465 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 86200.1us
I20240426 09:55:59.274694 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8574d769068f477e859574c959d5880e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 09:55:59.285480 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 46259.2us
I20240426 09:55:59.299425 10618 raft_consensus.cc:1203] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 8 LEARNER]: Deduplicated request from leader. Original: 8.5->[8.6-8.6] Dedup: 8.6->[]
I20240426 09:55:59.327716 10013 raft_consensus.cc:1261] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 09:55:59.332834 12757 consensus_queue.cc:1035] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:59.334160 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:59.350545 9936 raft_consensus.cc:1261] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 09:55:59.352123 12757 consensus_queue.cc:1035] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:55:59.479983 12770 tablet_copy_client.cc:783] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:55:59.480552 12770 tablet_copy_client.cc:647] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:55:59.484818 10612 raft_consensus.cc:1203] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 8 LEARNER]: Deduplicated request from leader. Original: 8.7->[8.8-8.8] Dedup: 8.8->[]
I20240426 09:55:59.500454 12770 tablet_copy_client.cc:523] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:55:59.515578 12770 tablet_bootstrap.cc:492] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:55:59.525228 12784 tablet_bootstrap.cc:492] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:59.557260 12784 tablet_bootstrap.cc:492] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:59.559549 12784 ts_tablet_manager.cc:1367] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.324s user 0.055s sys 0.047s
I20240426 09:55:59.615227 10007 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:55:59.616425 10007 raft_consensus.cc:595] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:59.627128 10089 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:59.628053 10089 raft_consensus.cc:595] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:55:59.642978 10089 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:55:59.643945 10089 raft_consensus.cc:595] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:59.675511 12704 raft_consensus.cc:979] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:59.677547 12757 raft_consensus.cc:1067] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:59.680869 9936 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:46580
I20240426 09:55:59.681460 9936 raft_consensus.cc:595] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:55:59.684145 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 09:55:59.687844 10013 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:55:59.688642 10013 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:59.689219 10013 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:55:59.698469 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:55:59.699370 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:55:59.708375 12784 raft_consensus.cc:348] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:59.715909 12784 raft_consensus.cc:729] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:59.719815 12784 consensus_queue.cc:260] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:59.757483 12801 raft_consensus.cc:979] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:59.777096 12801 raft_consensus.cc:1067] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:55:59.741989 12784 ts_tablet_manager.cc:1398] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.182s user 0.007s sys 0.006s
I20240426 09:55:59.779018 10007 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34390
I20240426 09:55:59.788609 10007 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:59.730999 10013 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:59.788978 10007 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:59.790866 10026 tablet_copy_service.cc:342] P 826f03b2fb9c4b9483623b041a1bf80e: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-69700f6b120b4f4ebc0b43b14166f4c9 received from {username='slave'} at 127.0.0.1:36762
I20240426 09:55:59.791381 10026 tablet_copy_service.cc:433] P 826f03b2fb9c4b9483623b041a1bf80e: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-69700f6b120b4f4ebc0b43b14166f4c9 on tablet 69700f6b120b4f4ebc0b43b14166f4c9 with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:59.799985 10007 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:59.818197 10013 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:55:59.822626 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:55:59.823396 10089 raft_consensus.cc:3032] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Stepping down as leader of term 8
I20240426 09:55:59.823698 10089 raft_consensus.cc:729] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:55:59.824501 10089 consensus_queue.cc:260] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:59.831755 10089 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:55:59.848875 10089 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 9.
I20240426 09:55:59.850517 9979 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:55:59.851485 12642 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election won for term 9
I20240426 09:55:59.852567 12642 raft_consensus.cc:686] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 9 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:59.853607 12642 consensus_queue.cc:237] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:55:59.862289 12770 tablet_bootstrap.cc:492] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:55:59.863221 12770 tablet_bootstrap.cc:492] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68: Bootstrap complete.
I20240426 09:55:59.864090 12770 ts_tablet_manager.cc:1367] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68: Time spent bootstrapping tablet: real 0.349s user 0.027s sys 0.012s
I20240426 09:55:59.867166 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:59.867941 9940 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:55:59.869728 12770 raft_consensus.cc:348] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 6 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:59.870684 12770 raft_consensus.cc:729] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 6 LEARNER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Initialized, Role: LEARNER
I20240426 09:55:59.871547 12770 consensus_queue.cc:260] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:55:59.872787 10007 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:55:59.876916 9940 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 9.
I20240426 09:55:59.877067 12770 ts_tablet_manager.cc:1398] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68: Time spent starting tablet: real 0.013s user 0.001s sys 0.005s
I20240426 09:55:59.878598 10026 tablet_copy_service.cc:342] P 826f03b2fb9c4b9483623b041a1bf80e: Request end of tablet copy session 1fee3da01ebc463396397057fdb81a68-8574d769068f477e859574c959d5880e received from {username='slave'} at 127.0.0.1:36762
I20240426 09:55:59.878822 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: true dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:55:59.879081 10026 tablet_copy_service.cc:433] P 826f03b2fb9c4b9483623b041a1bf80e: ending tablet copy session 1fee3da01ebc463396397057fdb81a68-8574d769068f477e859574c959d5880e on tablet 8574d769068f477e859574c959d5880e with peer 1fee3da01ebc463396397057fdb81a68
I20240426 09:55:59.879384 10612 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:59.891374 12755 raft_consensus.cc:1050] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
W20240426 09:55:59.891897 12755 raft_consensus.cc:1054] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e: Unable to promote non-voter 1fee3da01ebc463396397057fdb81a68: Service unavailable: leader transfer in progress
I20240426 09:55:59.893019 9836 catalog_manager.cc:5484] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 8 to 9, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 9 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:55:59.896471 10612 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:55:59.897751 9980 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:55:59.898591 12642 raft_consensus.cc:2781] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:55:59.899104 12642 raft_consensus.cc:686] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:55:59.900012 12642 consensus_queue.cc:237] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:59.907023 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:55:59.907644 9934 raft_consensus.cc:3032] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:55:59.907925 9934 raft_consensus.cc:729] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:55:59.908797 9934 consensus_queue.cc:260] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:55:59.910096 9934 raft_consensus.cc:3037] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:55:59.917248 9934 raft_consensus.cc:2454] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:55:59.921303 12755 raft_consensus.cc:979] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:59.923398 12755 raft_consensus.cc:1067] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Signalling peer f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:55:59.933972 12817 raft_consensus.cc:1050] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:55:59.995790 9934 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:47242
I20240426 09:55:59.996266 9934 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:55:59.996500 9934 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
W20240426 09:56:00.003154 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 50437.7us
I20240426 09:56:00.006990 12816 raft_consensus.cc:979] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee: : Instructing follower f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:56:00.007596 12816 raft_consensus.cc:1067] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Signalling peer f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:56:00.009168 9940 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:47248
I20240426 09:56:00.009853 9940 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:56:00.010187 9940 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:56:00.011353 9934 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:00.015548 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:00.016060 10014 raft_consensus.cc:3032] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Stepping down as leader of term 8
I20240426 09:56:00.016355 10014 raft_consensus.cc:729] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:00.017061 10014 consensus_queue.cc:260] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:00.018510 10014 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:00.019968 9934 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:00.021832 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:00.022384 10081 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:00.028041 12817 consensus_queue.cc:237] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 4, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:00.035698 10081 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 9.
I20240426 09:56:00.035221 9940 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:00.047631 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:00.048162 10612 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:56:00.049106 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:00.050493 9904 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:00.049657 10081 raft_consensus.cc:3032] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:56:00.054852 9940 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:00.055119 10081 raft_consensus.cc:729] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:00.054396 10612 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:56:00.059566 12815 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 09:56:00.062342 12815 raft_consensus.cc:686] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 9 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:00.062820 9905 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:00.063741 12801 raft_consensus.cc:2781] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 09:56:00.061882 10081 consensus_queue.cc:260] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:00.063274 12815 consensus_queue.cc:237] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:00.064218 12801 raft_consensus.cc:686] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:00.074025 12801 consensus_queue.cc:237] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:00.080798 10014 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 9.
I20240426 09:56:00.088864 10089 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240426 09:56:00.091918 12690 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 09:56:00.100829 10081 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:56:00.112426 10612 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 8 LEARNER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240426 09:56:00.135887 10081 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:56:00.144870 12690 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
W20240426 09:56:00.178243 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 75617.7us
I20240426 09:56:00.182722 9940 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240426 09:56:00.176497 9836 catalog_manager.cc:5484] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 7 to 8, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 8 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:56:00.197751 12755 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:56:00.234109 9835 catalog_manager.cc:5484] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 8 to 9, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 9 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 09:56:00.248685 12690 raft_consensus.cc:2932] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Committing config change with OpId 8.7: config changed from index 6 to 7, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:00.250514 10618 raft_consensus.cc:2932] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Committing config change with OpId 8.7: config changed from index 6 to 7, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:00.251600 9940 raft_consensus.cc:2932] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Committing config change with OpId 8.7: config changed from index 6 to 7, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:00.273532 10618 raft_consensus.cc:1203] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Deduplicated request from leader. Original: 7.6->[7.7-7.7] Dedup: 7.7->[]
I20240426 09:56:00.279436 10081 raft_consensus.cc:1261] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240426 09:56:00.279424 9838 catalog_manager.cc:5484] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index 6 to 7, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 8 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:56:00.283730 12642 consensus_queue.cc:1035] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20240426 09:56:00.265734 10082 raft_consensus.cc:2932] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Committing config change with OpId 8.7: config changed from index 6 to 7, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:00.323127 9835 catalog_manager.cc:5484] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 3 to 4, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 4 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:56:00.328828 9940 raft_consensus.cc:1261] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240426 09:56:00.331621 9939 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47232
I20240426 09:56:00.332177 9939 raft_consensus.cc:595] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 9 LEADER]: Received request to transfer leadership
I20240426 09:56:00.331471 12690 consensus_queue.cc:1035] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 09:56:00.342082 10007 consensus_queue.cc:237] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:00.353199 10618 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 09:56:00.355185 12690 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 09:56:00.358042 9940 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 09:56:00.362036 12690 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 09:56:00.364955 12755 raft_consensus.cc:2932] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Committing config change with OpId 8.8: config changed from index 7 to 8, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:00.371232 10618 raft_consensus.cc:2932] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Committing config change with OpId 8.8: config changed from index 7 to 8, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:00.382423 9935 raft_consensus.cc:2932] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Committing config change with OpId 8.8: config changed from index 7 to 8, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:00.415745 10612 raft_consensus.cc:1203] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 6 LEARNER]: Deduplicated request from leader. Original: 6.4->[6.5-6.5] Dedup: 6.5->[]
I20240426 09:56:00.417841 9838 catalog_manager.cc:5484] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 7 to 8, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New cstate: current_term: 8 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:00.420531 9825 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5e80ebfd190643a99ef02b6ea2876426 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:56:00.429559 9935 raft_consensus.cc:1261] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 09:56:00.430909 12817 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 09:56:00.438419 10077 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e80ebfd190643a99ef02b6ea2876426 with delete_type TABLET_DATA_TOMBSTONED (TS d3a480e831d649139ff016fb68aee3ee not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:59730
I20240426 09:56:00.440538 10618 raft_consensus.cc:1261] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 09:56:00.441859 12690 consensus_queue.cc:1035] T fb385fda44e94153ae3cd9ca21240939 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 09:56:00.451224 9940 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47232
I20240426 09:56:00.451741 9940 raft_consensus.cc:595] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 9 LEADER]: Received request to transfer leadership
W20240426 09:56:00.452917 9826 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 3dc043e5f7b241afb7119ba8b06ed37f already in progress [suppressed 11 similar messages]
I20240426 09:56:00.475585 12838 tablet_replica.cc:330] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee: stopping tablet replica
I20240426 09:56:00.477756 12838 raft_consensus.cc:2229] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Raft consensus shutting down.
I20240426 09:56:00.479746 12838 raft_consensus.cc:2258] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Raft consensus is shut down!
I20240426 09:56:00.481802 9935 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47232
I20240426 09:56:00.482376 9935 raft_consensus.cc:595] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 9 LEADER]: Received request to transfer leadership
W20240426 09:56:00.483937 9887 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 3dc043e5f7b241afb7119ba8b06ed37f already in progress
W20240426 09:56:00.484248 9887 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 3dc043e5f7b241afb7119ba8b06ed37f already in progress
I20240426 09:56:00.485157 12838 ts_tablet_manager.cc:1875] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:56:00.497402 10014 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 09:56:00.514026 12828 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 09:56:00.532764 10084 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 09:56:00.538208 12828 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 09:56:00.553802 10612 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 8 LEARNER]: Advancing to term 9
I20240426 09:56:00.565155 10612 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 9 LEARNER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 09:56:00.566745 12828 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 09:56:00.579737 12829 raft_consensus.cc:979] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9: : Instructing follower d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:56:00.580240 12815 raft_consensus.cc:1067] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 9 LEADER]: Signalling peer d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:56:00.581753 10084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:59762
I20240426 09:56:00.582350 10084 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:56:00.582796 10084 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:00.612948 10084 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:00.613883 12828 raft_consensus.cc:1050] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
W20240426 09:56:00.614368 12828 raft_consensus.cc:1054] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9: Unable to promote non-voter 1fee3da01ebc463396397057fdb81a68: Service unavailable: leader transfer in progress
I20240426 09:56:00.615274 10084 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:00.616127 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:00.617743 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:00.618326 10007 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:00.619390 9935 raft_consensus.cc:3032] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 09:56:00.619760 9935 raft_consensus.cc:729] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:00.621096 9935 consensus_queue.cc:260] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:00.623173 10618 raft_consensus.cc:1261] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 09:56:00.625257 12829 consensus_queue.cc:1035] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:56:00.637676 9935 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:00.637923 10007 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 10.
I20240426 09:56:00.642344 10055 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:00.643177 12704 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader election won for term 10
I20240426 09:56:00.643708 12704 raft_consensus.cc:686] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 10 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:00.644518 12704 consensus_queue.cc:237] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:00.645493 10012 raft_consensus.cc:1226] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 9. Current term is 10. Ops: []
I20240426 09:56:00.657219 10084 raft_consensus.cc:1261] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 09:56:00.658706 12828 consensus_queue.cc:1035] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:56:00.663513 12838 ts_tablet_manager.cc:1888] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 8.7
I20240426 09:56:00.664029 12838 log.cc:1196] T 5e80ebfd190643a99ef02b6ea2876426 P d3a480e831d649139ff016fb68aee3ee: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-2-root/wals/5e80ebfd190643a99ef02b6ea2876426
I20240426 09:56:00.668247 9838 catalog_manager.cc:5484] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 9 to 10, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 10 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 09:56:00.677013 9823 catalog_manager.cc:4830] TS d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): tablet 5e80ebfd190643a99ef02b6ea2876426 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:56:00.696246 9935 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 10.
I20240426 09:56:00.737829 12642 raft_consensus.cc:1050] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:56:00.740285 12642 consensus_queue.cc:237] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:00.822034 12817 raft_consensus.cc:1050] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:56:00.824666 12817 consensus_queue.cc:237] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 3, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:00.860123 10612 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 7 LEARNER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 09:56:00.862154 12755 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 09:56:00.932956 10084 raft_consensus.cc:1261] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240426 09:56:00.945947 10612 raft_consensus.cc:1261] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 6 LEARNER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240426 09:56:00.935269 12755 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:56:00.957583 12690 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:56:00.976133 9934 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 09:56:00.977444 10084 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 09:56:00.979722 12690 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 09:56:00.980737 12642 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 09:56:00.981814 9935 raft_consensus.cc:1261] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240426 09:56:00.984982 12690 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:56:01.008777 12817 raft_consensus.cc:2932] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:01.018610 10618 raft_consensus.cc:2932] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:01.007787 12642 raft_consensus.cc:2932] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Committing config change with OpId 7.8: config changed from index 7 to 8, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:01.014539 9934 raft_consensus.cc:2932] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:01.043211 9939 raft_consensus.cc:2932] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Committing config change with OpId 7.8: config changed from index 7 to 8, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:01.046471 10082 raft_consensus.cc:2932] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:01.048763 10084 raft_consensus.cc:2932] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Committing config change with OpId 7.8: config changed from index 7 to 8, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:01.061952 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:56:01.070719 9838 catalog_manager.cc:5484] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 reported cstate change: config changed from index 5 to 6, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } attrs { replace: true } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:01.084784 10014 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240426 09:56:01.101570 9835 catalog_manager.cc:5484] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 reported cstate change: config changed from index 7 to 8, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:01.115782 12704 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 09:56:01.146023 10007 consensus_queue.cc:237] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:01.148931 10612 raft_consensus.cc:2932] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Committing config change with OpId 7.8: config changed from index 7 to 8, 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:01.185424 10618 raft_consensus.cc:1261] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20240426 09:56:01.190423 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 47688.1us
I20240426 09:56:01.196859 9934 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240426 09:56:01.200079 10084 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:56:01.200662 10084 raft_consensus.cc:595] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 10 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:01.203132 10618 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 9 LEARNER]: Advancing to term 10
W20240426 09:56:01.227734 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 102840us
I20240426 09:56:01.229748 9934 raft_consensus.cc:1261] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 09:56:01.217701 12755 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20240426 09:56:01.211611 12855 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20240426 09:56:01.252513 10618 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 10 LEARNER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240426 09:56:01.260058 12816 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 09:56:01.261633 12690 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:56:01.265838 10012 consensus_queue.cc:237] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:01.300311 9934 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
mode: GRACEFUL
new_leader_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:46580
I20240426 09:56:01.300945 9934 raft_consensus.cc:595] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Received request to transfer leadership to TS d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:01.307291 10014 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:56:01.307844 10014 raft_consensus.cc:595] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Received request to transfer leadership to TS d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:01.318224 10015 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
mode: GRACEFUL
new_leader_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:34938
I20240426 09:56:01.318789 10015 raft_consensus.cc:595] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Received request to transfer leadership to TS f12521b6d51242629cf4297864df00b9
I20240426 09:56:01.323355 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 09:56:01.325542 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:56:01.328155 10612 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20240426 09:56:01.331328 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:56:01.345537 10084 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20240426 09:56:01.377758 12690 raft_consensus.cc:2932] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Committing config change with OpId 6.7: config changed from index 6 to 7, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:01.383889 12869 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.011s
I20240426 09:56:01.385921 12816 raft_consensus.cc:979] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:56:01.386417 12816 raft_consensus.cc:1067] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 10 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:56:01.393785 12855 raft_consensus.cc:1050] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
W20240426 09:56:01.394170 12855 raft_consensus.cc:1054] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee: Unable to promote non-voter 1fee3da01ebc463396397057fdb81a68: Service unavailable: leader transfer in progress
I20240426 09:56:01.402948 10015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:56:01.403594 10015 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:56:01.403950 10015 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:01.409914 12870 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 09:56:01.435730 10015 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:01.438230 12642 raft_consensus.cc:2932] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Committing config change with OpId 7.9: config changed from index 8 to 9, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:01.446372 10089 raft_consensus.cc:2932] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Committing config change with OpId 7.9: config changed from index 8 to 9, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:01.468910 9825 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8574d769068f477e859574c959d5880e with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:56:01.470983 10618 raft_consensus.cc:2932] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:01.472152 9935 raft_consensus.cc:2932] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:01.475440 9825 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 69700f6b120b4f4ebc0b43b14166f4c9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 09:56:01.476452 10015 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:01.477254 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:01.490969 10084 raft_consensus.cc:3032] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 10 LEADER]: Stepping down as leader of term 10
I20240426 09:56:01.491407 10084 raft_consensus.cc:729] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:01.482937 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:01.492200 9934 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:01.492695 9835 catalog_manager.cc:5484] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index 6 to 7, VOTER d3a480e831d649139ff016fb68aee3ee (127.8.141.131) evicted. New cstate: current_term: 6 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:56:01.497112 9835 catalog_manager.cc:5484] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: config changed from index 8 to 9, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New cstate: current_term: 7 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:56:01.503852 9934 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 11.
I20240426 09:56:01.511179 9940 raft_consensus.cc:1226] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 10. Current term is 11. Ops: []
I20240426 09:56:01.514425 10618 raft_consensus.cc:2932] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Committing config change with OpId 7.9: config changed from index 8 to 9, VOTER f12521b6d51242629cf4297864df00b9 (127.8.141.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:01.492322 10084 consensus_queue.cc:260] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:01.517396 10084 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:01.528556 9982 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:01.530717 12690 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Leader election won for term 11
I20240426 09:56:01.531193 12690 raft_consensus.cc:686] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 11 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:01.532021 12690 consensus_queue.cc:237] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:01.533035 10084 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 11.
I20240426 09:56:01.550021 10077 tablet_service.cc:1514] Processing DeleteTablet for tablet 8574d769068f477e859574c959d5880e with delete_type TABLET_DATA_TOMBSTONED (TS d3a480e831d649139ff016fb68aee3ee not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59730
I20240426 09:56:01.550460 9925 tablet_service.cc:1514] Processing DeleteTablet for tablet 69700f6b120b4f4ebc0b43b14166f4c9 with delete_type TABLET_DATA_TOMBSTONED (TS f12521b6d51242629cf4297864df00b9 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47232
I20240426 09:56:01.565191 9836 catalog_manager.cc:5484] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 10 to 11, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 11 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 09:56:01.573884 12873 tablet_replica.cc:330] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee: stopping tablet replica
I20240426 09:56:01.579236 12873 raft_consensus.cc:2229] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Raft consensus shutting down.
I20240426 09:56:01.580078 12873 raft_consensus.cc:2258] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Raft consensus is shut down!
I20240426 09:56:01.583523 12874 tablet_replica.cc:330] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9: stopping tablet replica
I20240426 09:56:01.585516 12873 ts_tablet_manager.cc:1875] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:56:01.601716 12873 ts_tablet_manager.cc:1888] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.6
I20240426 09:56:01.602128 12873 log.cc:1196] T 8574d769068f477e859574c959d5880e P d3a480e831d649139ff016fb68aee3ee: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-2-root/wals/8574d769068f477e859574c959d5880e
I20240426 09:56:01.603876 9823 catalog_manager.cc:4830] TS d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): tablet 8574d769068f477e859574c959d5880e (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:56:01.608310 12856 raft_consensus.cc:979] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:56:01.608759 12690 raft_consensus.cc:1067] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Signalling peer f12521b6d51242629cf4297864df00b9 to start an election
I20240426 09:56:01.614672 12874 raft_consensus.cc:2229] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Raft consensus shutting down.
I20240426 09:56:01.615527 12874 raft_consensus.cc:2258] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240426 09:56:01.619216 12874 ts_tablet_manager.cc:1875] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 09:56:01.631975 9940 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
from {username='slave'} at 127.0.0.1:47242
I20240426 09:56:01.632571 9940 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:56:01.632944 9940 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:01.634652 12874 ts_tablet_manager.cc:1888] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.8
I20240426 09:56:01.635174 12874 log.cc:1196] T 69700f6b120b4f4ebc0b43b14166f4c9 P f12521b6d51242629cf4297864df00b9: Deleting WAL directory at /tmp/dist-test-taskrNsLxW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714125217154154-8758-0/minicluster-data/ts-0-root/wals/69700f6b120b4f4ebc0b43b14166f4c9
I20240426 09:56:01.636734 9826 catalog_manager.cc:4830] TS f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): tablet 69700f6b120b4f4ebc0b43b14166f4c9 (table test-workload [id=423e8fc5d5d743c08b5472616da5df50]) successfully deleted
I20240426 09:56:01.644160 9940 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:01.646569 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:01.647214 10015 raft_consensus.cc:3032] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:56:01.647547 10015 raft_consensus.cc:729] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:01.648272 10015 consensus_queue.cc:260] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:01.650550 10015 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:01.658041 10015 raft_consensus.cc:2454] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:56:01.659164 9906 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:01.660324 12801 raft_consensus.cc:2781] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:56:01.660804 12801 raft_consensus.cc:686] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:01.661613 12801 consensus_queue.cc:237] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:01.663769 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:01.664415 10618 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:01.679298 10618 raft_consensus.cc:2454] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:56:01.680716 9940 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:01.691552 9836 catalog_manager.cc:5484] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 4 to 5, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 5 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:56:01.712602 12869 raft_consensus.cc:979] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e: : Instructing follower d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:56:01.713147 12869 raft_consensus.cc:1067] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Signalling peer d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:56:01.714949 10089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:59746
I20240426 09:56:01.715466 10089 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:56:01.715797 10089 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:01.733633 10089 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:01.740825 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:01.741436 10014 raft_consensus.cc:3032] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:56:01.741746 10014 raft_consensus.cc:729] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:01.742403 10014 consensus_queue.cc:260] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:01.743505 10014 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:01.743989 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:01.744539 10618 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:01.749619 10014 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:56:01.750758 10055 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:01.751561 12816 raft_consensus.cc:2781] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:56:01.753584 12816 raft_consensus.cc:686] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:01.754423 12816 consensus_queue.cc:237] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:01.760298 10618 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:56:01.762710 10089 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:01.791734 9836 catalog_manager.cc:5484] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 4 to 5, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 5 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:56:01.815832 12801 raft_consensus.cc:979] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9: : Instructing follower d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:56:01.816344 12801 raft_consensus.cc:1067] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Signalling peer d3a480e831d649139ff016fb68aee3ee to start an election
I20240426 09:56:01.817817 10089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:59762
I20240426 09:56:01.818398 10089 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:56:01.818738 10089 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:01.850560 10089 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:01.853178 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:01.853737 9940 raft_consensus.cc:3032] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:56:01.854063 9940 raft_consensus.cc:729] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:01.854813 9940 consensus_queue.cc:260] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:01.855855 9940 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:01.859738 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:01.860323 10618 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:01.877144 10618 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:56:01.879422 10054 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:01.880350 12816 raft_consensus.cc:2781] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:56:01.880882 12816 raft_consensus.cc:686] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:01.881781 12816 consensus_queue.cc:237] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:01.859678 10089 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:01.971987 10089 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20240426 09:56:02.003661 12858 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 09:56:02.065238 10618 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 10 LEARNER]: Advancing to term 11
W20240426 09:56:02.065642 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 131063us
I20240426 09:56:02.080724 10618 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 11 LEARNER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20240426 09:56:02.091058 12869 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.002s
I20240426 09:56:02.093180 9935 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
W20240426 09:56:02.121443 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 108039us
I20240426 09:56:02.136567 12856 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 09:56:03.010249 12884 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:03.010924 12884 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.013257 12884 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.082372 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1879) took 1015 ms. Trace:
I20240426 09:56:03.083002 9934 rpcz_store.cc:276] 0426 09:56:02.066506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:02.183784 (+117278us) service_pool.cc:225] Handling call
0426 09:56:03.081816 (+898032us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:03.082000 (+ 184us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:03.082024 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:03.082065 (+ 41us) raft_consensus.cc:1631] Marking committed up to 9
0426 09:56:03.082104 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:03.082154 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:03.082283 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:03.105677 12882 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:03.211287 12882 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.213615 12882 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.215047 12882 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:03.216457 12882 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.209429 12888 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.218179 12888 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.220436 12888 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:03.114607 12883 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:03.221208 12883 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.223412 12883 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:03.118507 12887 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:03.224177 12887 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.217461 12890 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:03.226897 12890 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.228335 12882 leader_election.cc:288] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.228817 12887 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.230020 12890 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.230782 12890 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.232118 12890 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.235042 12890 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:03.239151 9940 consensus_meta.cc:287] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 1.383s user 0.023s sys 0.013s
I20240426 09:56:03.239585 9940 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:56:03.240175 9940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 2272) took 1387 ms. Trace:
I20240426 09:56:03.240522 9940 rpcz_store.cc:276] 0426 09:56:01.852651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:01.852971 (+ 320us) service_pool.cc:225] Handling call
0426 09:56:03.240102 (+1387131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:03.266420 12887 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:03.267035 12887 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:03.268055 10613 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.153s user 0.019s sys 0.015s
W20240426 09:56:03.268424 10613 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.154s user 0.020s sys 0.016s
I20240426 09:56:03.269034 12887 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:03.270061 12887 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.270655 12887 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.271759 10613 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45564 (request call id 562) took 1176 ms. Trace:
I20240426 09:56:03.273043 12887 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.272150 10613 rpcz_store.cc:276] 0426 09:56:02.095386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:02.095530 (+ 144us) service_pool.cc:225] Handling call
0426 09:56:02.095823 (+ 293us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:03.268727 (+1172904us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:03.268767 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:03.269188 (+ 421us) log.cc:842] Serialized 26 byte log entry
0426 09:56:03.269983 (+ 795us) raft_consensus.cc:1631] Marking committed up to 10
0426 09:56:03.270016 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:03.270085 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:03.271500 (+ 1415us) raft_consensus.cc:1671] finished
0426 09:56:03.271517 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:03.271673 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18281,"raft.queue_time_us":21253,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"thread_start_us":1154702,"threads_started":2,"wal-append.queue_time_us":854}
I20240426 09:56:03.276604 12887 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
W20240426 09:56:03.278149 12869 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.139s user 0.008s sys 0.011s
I20240426 09:56:03.278020 12887 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:03.278499 12869 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.139s user 0.009s sys 0.011s
I20240426 09:56:03.279946 12887 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:03.285387 12893 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.286082 12893 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:03.285595 9935 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.093s user 0.002s sys 0.007s
W20240426 09:56:03.287026 9935 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.094s user 0.003s sys 0.007s
I20240426 09:56:03.289336 12893 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.291683 12869 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:03.292227 12869 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.292822 12892 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:03.293355 12892 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.294068 12869 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.294747 12893 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.295310 12892 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.295260 12893 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.301002 12893 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.304114 9836 catalog_manager.cc:5484] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 4 to 5, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 5 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:56:03.305310 12869 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.315047 12869 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:03.318219 9979 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.180s user 0.010s sys 0.007s
I20240426 09:56:03.318188 12869 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:03.318598 9979 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.180s user 0.010s sys 0.007s
W20240426 09:56:03.319408 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 1.18096e+06us
I20240426 09:56:03.306388 12893 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:03.320087 12893 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.321319 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1888) took 1152 ms. Trace:
I20240426 09:56:03.321956 12893 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.323182 12893 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.321676 9935 rpcz_store.cc:276] 0426 09:56:02.169241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:02.191239 (+ 21998us) service_pool.cc:225] Handling call
0426 09:56:02.191528 (+ 289us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:02.191871 (+ 343us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 09:56:03.287390 (+1095519us) log.cc:842] Serialized 14 byte log entry
0426 09:56:03.306985 (+ 19595us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:03.307415 (+ 430us) log.cc:842] Serialized 26 byte log entry
0426 09:56:03.307548 (+ 133us) raft_consensus.cc:1631] Marking committed up to 10
0426 09:56:03.307581 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:03.307660 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:03.321102 (+ 13442us) raft_consensus.cc:1671] finished
0426 09:56:03.321121 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:03.321239 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":893678,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":41,"raft.queue_time_us":12378,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":211,"spinlock_wait_cycles":1316608,"thread_start_us":1113907,"threads_started":2,"wal-append.queue_time_us":19268}
I20240426 09:56:03.323830 12893 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.325733 12893 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.318960 12898 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.327421 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 2281) took 1142 ms. Trace:
I20240426 09:56:03.327519 12898 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.327890 9932 rpcz_store.cc:276] 0426 09:56:02.184598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:03.326502 (+1141904us) service_pool.cc:225] Handling call
0426 09:56:03.326854 (+ 352us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:03.327035 (+ 181us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:03.327056 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:03.327090 (+ 34us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:56:03.327122 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:03.327166 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:03.327273 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:03.328955 12887 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.329576 12898 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.329401 12887 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.331346 12887 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:03.332088 12898 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.334268 12898 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.336050 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:03.337314 12690 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:03.337785 12690 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.339424 12690 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.339649 12829 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.340230 12829 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.342068 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:03.343943 12879 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.344281 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:03.344483 12879 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.347308 12878 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:03.347815 12878 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.348337 12879 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.348866 12879 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:03.349288 12879 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.350109 12878 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:03.351147 12879 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:56:03.342626 10584 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.228s user 0.011s sys 0.015s
W20240426 09:56:03.351907 10584 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.237s user 0.012s sys 0.015s
I20240426 09:56:03.353540 12856 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:03.354036 12856 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:03.355018 10055 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.216s user 0.000s sys 0.021s
I20240426 09:56:03.355396 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:03.354966 10611 raft_consensus.cc:1261] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 09:56:03.342227 12829 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.362057 12898 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.364856 12879 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:03.365904 12889 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:03.366423 12889 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.365576 12879 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.368258 12889 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.370592 12883 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:03.371126 12883 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.372906 12883 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.373328 12883 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:03.373742 12883 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.370095 12879 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.365510 12888 consensus_queue.cc:1035] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20240426 09:56:03.377993 12889 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:03.380700 12889 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:03.425427 8763 debug-util.cc:398] Leaking SignalData structure 0x7b080000eb00 after lost signal to thread 9979
W20240426 09:56:03.444373 10583 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.330s user 0.010s sys 0.010s
W20240426 09:56:03.444769 10583 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.330s user 0.011s sys 0.011s
I20240426 09:56:03.384070 12889 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:03.501646 9870 debug-util.cc:398] Leaking SignalData structure 0x7b0800328f60 after lost signal to thread 8761
I20240426 09:56:03.507802 12878 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.508564 12878 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.509100 12871 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.509657 12871 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.511649 12871 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.513056 12896 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.513563 12896 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.515704 12896 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.516286 12896 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.516777 12896 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.519191 12896 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.517524 12829 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.521240 12883 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:03.571282 12893 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.571923 12893 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.573972 12893 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.583756 12893 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.584378 12893 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.586951 12893 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:03.592917 12858 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.452s user 0.010s sys 0.014s
W20240426 09:56:03.593469 12858 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.452s user 0.010s sys 0.014s
I20240426 09:56:03.595450 12898 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.596117 12898 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.598762 12895 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.599344 12895 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.601305 12895 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.601764 12895 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.602195 12895 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.604251 12895 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.347676 12887 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.625761 12887 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.627853 12887 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.628528 12896 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.629168 12896 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.631770 12896 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.521682 12829 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.634001 12829 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.519670 12871 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:03.636042 12871 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.560232 12859 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:03.638568 12859 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.641345 12859 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.641832 12859 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.646219 12898 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.649648 9941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 2280) took 1465 ms. Trace:
I20240426 09:56:03.650055 9941 rpcz_store.cc:276] 0426 09:56:02.184332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:03.648752 (+1464420us) service_pool.cc:225] Handling call
0426 09:56:03.649080 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:03.649240 (+ 160us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:03.649263 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:03.649303 (+ 40us) raft_consensus.cc:1631] Marking committed up to 9
0426 09:56:03.649348 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:03.649393 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:03.649494 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:03.699358 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:56:03.797506 12883 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.800137 12883 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.802225 12883 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.814788 12882 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.815371 12882 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.817432 12882 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.817910 12882 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.818325 12882 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.820219 12882 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.618683 12878 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:56:03.872880 9870 debug-util.cc:398] Leaking SignalData structure 0x7b0800328d20 after lost signal to thread 9873
I20240426 09:56:03.892551 12856 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.904034 12887 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:03.904623 12887 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.911824 12889 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.915143 12893 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.913022 12894 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.915784 12893 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.916096 12894 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.914973 12889 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.917549 12829 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.918025 12894 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.918136 12829 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.919935 12893 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.920323 12829 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.918736 12889 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:03.921082 12871 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.925109 12882 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.925588 12882 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.927055 12890 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:03.927532 12890 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.921363 12889 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:03.929385 12890 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.929179 12889 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.931142 12882 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:03.933493 9904 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.036s sys 0.020s
W20240426 09:56:03.933831 9904 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.596s user 0.036s sys 0.020s
I20240426 09:56:03.934316 12882 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:03.939183 12871 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.939680 12871 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.941485 12871 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.941946 12871 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.943291 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:03.944378 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:03.947789 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:03.948493 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:03.948782 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:03.950003 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:03.950402 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:03.942380 12871 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.948377 9934 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:56:03.952538 12871 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.956765 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:03.913120 12892 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.957527 9941 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:56:03.952901 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:03.957893 12892 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:03.958482 10087 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:03.931156 12889 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.913178 12898 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:03.960249 12898 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.962356 12898 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.959487 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:03.913256 12883 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:03.963335 12883 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:03.966346 10087 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:56:03.966120 9904 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:03.966873 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:03.948299 12896 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.967432 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:03.969846 12883 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:03.970254 10087 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 8.
I20240426 09:56:03.970480 12893 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:03.968732 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:03.970968 12893 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:03.968390 12896 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.971378 12893 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:03.973776 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:03.974152 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:03.975355 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:03.975473 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:03.975988 9936 raft_consensus.cc:2454] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:56:03.971319 10082 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:03.976851 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:03.976465 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:03.973716 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:03.979068 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:03.980219 12896 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.981314 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:03.981891 10087 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 9.
I20240426 09:56:03.906733 12887 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.985656 12892 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:03.642282 12859 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:03.988137 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:03.988823 12859 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:03.988787 10620 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:56:03.989902 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:03.993403 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:03.993954 10087 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:03.996373 12882 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.001399 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.002032 10617 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:56:04.002480 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.003086 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.004040 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.004415 12882 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.005295 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.006090 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.002025 9836 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50254 (request call id 321) took 2047 ms. Trace:
I20240426 09:56:04.006705 10611 raft_consensus.cc:2454] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:56:04.007539 12869 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:04.007663 12690 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:04.008111 12869 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.008422 12910 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:04.008040 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.006481 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.009213 12910 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.010082 12869 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:04.010954 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 640455us
I20240426 09:56:04.011678 12910 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.012207 12910 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
W20240426 09:56:04.012650 10585 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.893s user 0.010s sys 0.011s
W20240426 09:56:04.013083 10585 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.894s user 0.011s sys 0.012s
I20240426 09:56:04.012773 12910 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.012020 10616 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 9.
I20240426 09:56:04.014858 12910 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.015012 12879 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.015395 12910 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:04.011687 10620 raft_consensus.cc:1261] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 09:56:04.015862 12910 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.015575 12879 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.006729 9836 rpcz_store.cc:276] 0426 09:56:01.954743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:01.954911 (+ 168us) service_pool.cc:225] Handling call
0426 09:56:04.001964 (+2047053us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:56:03.330714 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:56:03.331226 (+ 512us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:56:03.331273 (+ 47us) write_op.cc:427] Acquired schema lock
0426 09:56:03.331290 (+ 17us) tablet.cc:660] Decoding operations
0426 09:56:03.331592 (+ 302us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:56:03.337120 (+ 5528us) write_op.cc:633] Partition lock acquired for write op
0426 09:56:03.337161 (+ 41us) tablet.cc:683] Acquiring locks for 1 operations
0426 09:56:03.363084 (+ 25923us) tablet.cc:699] Row locks acquired
0426 09:56:03.363112 (+ 28us) write_op.cc:260] PREPARE: finished
0426 09:56:03.363296 (+ 184us) write_op.cc:270] Start()
0426 09:56:03.366400 (+ 3104us) write_op.cc:276] Timestamp: P: 1714125363363240 usec, L: 0
0426 09:56:03.366430 (+ 30us) op_driver.cc:331] REPLICATION: starting
0426 09:56:03.368675 (+ 2245us) log.cc:842] Serialized 536 byte log entry
0426 09:56:03.951093 (+582418us) op_driver.cc:444] REPLICATION: finished
0426 09:56:03.998117 (+ 47024us) write_op.cc:301] APPLY: starting
0426 09:56:03.998193 (+ 76us) tablet.cc:1365] starting BulkCheckPresence
0426 09:56:03.998281 (+ 88us) tablet.cc:1368] finished BulkCheckPresence
0426 09:56:03.998381 (+ 100us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:56:03.998574 (+ 193us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:56:03.998747 (+ 173us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0426 09:56:03.998823 (+ 76us) write_op.cc:312] APPLY: finished
0426 09:56:03.999005 (+ 182us) log.cc:842] Serialized 23 byte log entry
0426 09:56:03.999154 (+ 149us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:56:03.999238 (+ 84us) write_op.cc:446] Released schema lock
0426 09:56:03.999277 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0426 09:56:04.006900 (+ 7623us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:56:04.006930 (+ 30us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":43119,"apply.run_cpu_time_us":1735,"apply.run_wall_time_us":9030,"num_ops":1,"prepare.queue_time_us":14310,"prepare.run_cpu_time_us":4190,"prepare.run_wall_time_us":348035,"raft.queue_time_us":291349,"replication_time_us":584300,"thread_start_us":602742,"threads_started":4,"wal-append.queue_time_us":288710}]]}
I20240426 09:56:04.021219 12879 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.023339 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.008374 12890 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:04.010602 12869 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:04.025653 12890 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.026235 12869 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.009827 10617 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
W20240426 09:56:04.027637 10586 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.010s sys 0.001s
W20240426 09:56:04.027980 10586 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.684s user 0.010s sys 0.001s
I20240426 09:56:04.029462 12869 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.030026 12869 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:04.031055 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.031639 10015 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 5.
I20240426 09:56:04.008142 12690 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.027570 12890 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.031348 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.036604 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.038004 12890 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:04.038477 12890 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.039455 10054 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:04.039618 10616 raft_consensus.cc:1261] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 09:56:04.040336 12890 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.040915 12890 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:04.041085 12888 raft_consensus.cc:2781] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:04.041477 12888 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:04.041801 12888 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:04.042681 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.043834 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.044620 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.045251 10619 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:56:04.045368 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.046015 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.044929 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.047099 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.047586 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.034528 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.048708 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.033938 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.049536 9937 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 7.
I20240426 09:56:04.033839 12856 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:04.046577 10619 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 8.
I20240426 09:56:04.033375 12906 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:04.035943 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.051550 12908 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.036187 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.052067 12908 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.047726 10611 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
W20240426 09:56:04.051663 9870 debug-util.cc:398] Leaking SignalData structure 0x7b080031f3e0 after lost signal to thread 9965
W20240426 09:56:04.054198 9870 debug-util.cc:398] Leaking SignalData structure 0x7b08000390c0 after lost signal to thread 10040
I20240426 09:56:04.054121 12908 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:04.030481 12869 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.034902 12690 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.036728 12903 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:04.057296 10054 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:04.058218 12690 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:04.059602 12901 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:04.060119 12901 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.050170 12856 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.062587 12869 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.062904 12911 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:04.059147 12690 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.063349 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.064002 10083 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:56:04.064736 12690 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.063266 12903 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.057922 12882 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:04.064154 12911 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.066200 12882 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.067164 12903 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:04.051689 12906 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.063251 12888 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.069538 12906 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.017997 12910 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.070327 12888 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.063097 12869 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:04.076576 12869 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.074657 12888 raft_consensus.cc:2781] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:04.079099 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.079520 12888 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.079929 12888 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:04.081893 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.065681 12856 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.083042 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.083145 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.083477 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.082494 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.084203 9937 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:56:04.083849 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.085799 10085 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 11.
W20240426 09:56:04.086654 9870 debug-util.cc:398] Leaking SignalData structure 0x7b08001d2d40 after lost signal to thread 10114
I20240426 09:56:04.087074 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.087320 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.087975 9937 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:04.089280 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.089498 12869 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:04.087875 9870 debug-util.cc:398] Leaking SignalData structure 0x7b080036b2c0 after lost signal to thread 10644
I20240426 09:56:04.090489 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.091982 9904 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.092695 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:04.093505 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.094161 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.093017 9904 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.094787 9935 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:04.095201 9904 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.095593 12896 raft_consensus.cc:2781] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:56:04.095970 12896 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:04.096015 12900 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:04.096444 12896 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:04.096524 12900 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.096971 12900 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
W20240426 09:56:04.099514 12858 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.004s sys 0.005s
W20240426 09:56:04.099859 12858 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.506s user 0.004s sys 0.005s
I20240426 09:56:04.101511 10056 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.102320 10056 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.103233 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.103693 12900 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.089932 12901 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:03.355346 10055 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.216s user 0.000s sys 0.021s
I20240426 09:56:04.104894 12895 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:04.105319 12895 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.105686 12895 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:04.105964 12859 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:04.105187 12908 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:04.106653 12908 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.106868 12900 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.106127 9904 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.108049 12859 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.108748 12908 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:04.109086 9904 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.109833 12829 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:04.110203 12829 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.110558 12829 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:04.112377 12895 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:04.112947 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 1.97394e+06us
I20240426 09:56:04.113233 10054 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:04.114114 10054 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:04.115156 12864 raft_consensus.cc:2781] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:56:04.115561 12864 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:04.115892 12864 raft_consensus.cc:3037] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:56:04.117451 10586 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.119104 12889 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 09:56:04.119534 12889 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.119773 10015 raft_consensus.cc:1261] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 09:56:04.120225 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.120831 10012 raft_consensus.cc:2454] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 5.
I20240426 09:56:04.122025 12901 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:04.122423 12901 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:04.122815 12901 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:04.124349 12887 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:56:04.127080 12889 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Advancing to term 10
W20240426 09:56:04.127657 10583 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.681s user 0.005s sys 0.004s
W20240426 09:56:04.127960 10583 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.681s user 0.005s sys 0.004s
I20240426 09:56:04.068132 12882 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.133988 12882 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:04.134527 12882 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.135632 12879 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.136134 12879 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.136509 12882 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.108464 12859 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:04.068056 12911 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.115243 10054 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:04.147421 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.147648 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.106238 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.148324 9937 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 5.
I20240426 09:56:04.149495 10586 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.150599 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.059160 12890 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.134219 12864 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.151229 9904 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.132870 12888 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.108513 12900 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 09:56:04.148394 12879 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.139364 12858 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:04.152674 12911 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:04.169749 12858 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.170526 12888 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.143128 12903 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:04.171222 12903 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.170244 12911 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.160094 12916 raft_consensus.cc:2781] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:04.173252 12916 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:04.173617 12916 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:04.174469 12892 raft_consensus.cc:2781] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:04.174870 12892 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.175379 12892 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:04.175733 12895 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.174257 12901 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.177915 12901 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.178742 9905 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:04.171748 12858 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.160912 12883 raft_consensus.cc:2781] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:04.163100 12894 consensus_queue.cc:1035] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:56:04.179889 9905 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:04.163666 12889 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.164476 12915 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.144194 12829 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.165037 12890 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.166110 12908 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.167918 12864 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.180855 9905 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.190155 9982 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.174302 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.191447 10008 raft_consensus.cc:2454] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:56:04.187587 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.077086 12893 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.173070 12903 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.194629 12908 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:04.193612 12895 raft_consensus.cc:2781] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:04.199956 12895 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.186383 12896 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.200356 12895 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:04.201516 12896 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.194003 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.188244 12915 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.194427 12889 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.195955 12829 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.196339 12890 raft_consensus.cc:2781] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:04.204418 12915 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:04.185037 12871 raft_consensus.cc:2726] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 09:56:04.180914 12894 raft_consensus.cc:2726] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:04.198693 12916 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.205440 9936 raft_consensus.cc:1226] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 4. Current term is 5. Ops: []
I20240426 09:56:04.207048 12916 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.192238 10012 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:04.204730 12890 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:04.209985 12890 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:04.211182 9941 raft_consensus.cc:1226] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:04.181685 12883 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.208906 12914 raft_consensus.cc:1050] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e: attempting to promote NON_VOTER 1fee3da01ebc463396397057fdb81a68 to VOTER
I20240426 09:56:04.198940 12859 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.222751 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.226323 10012 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:56:04.230180 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.186091 12900 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.233566 9905 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.224923 12883 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:04.237432 9905 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:56:04.236693 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 121509us
I20240426 09:56:04.225986 12864 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:04.241082 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.210880 12895 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.243268 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.243882 10012 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:56:04.244612 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:04.235319 12911 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.246133 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.246368 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:04.236487 12894 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:04.241971 12816 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.244618 12871 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:04.236168 12900 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:04.238025 12893 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.252516 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:04.256367 10619 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:04.253087 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:04.253182 10585 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:04.251722 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.261245 10012 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:56:04.233297 12892 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.264176 10585 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:04.276605 10585 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:04.277499 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.277750 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.268214 12916 raft_consensus.cc:2726] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:04.250042 12894 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.278717 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.267063 10611 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:04.281994 10088 raft_consensus.cc:2379] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 10: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:04.284274 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.285665 12859 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:04.282953 12894 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:04.279343 12901 raft_consensus.cc:2781] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:04.290510 12901 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:04.290939 12901 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:04.290939 12903 raft_consensus.cc:2781] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:04.291450 12903 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:04.291781 12903 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:04.255682 12864 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.294726 12864 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.290640 10611 raft_consensus.cc:2454] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:56:04.300031 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:04.295148 12894 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.303349 10617 raft_consensus.cc:1226] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:04.299569 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.299274 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.304898 10090 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 5.
I20240426 09:56:04.305073 12892 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.300204 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:04.306455 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.307463 12895 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.308375 10054 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:04.308950 12894 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.309700 12914 consensus_queue.cc:237] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 10, Current term: 11, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.297961 12903 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.300544 10086 raft_consensus.cc:1226] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:04.301240 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.297860 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.300539 10611 raft_consensus.cc:1226] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 5. Current term is 6. Ops: []
I20240426 09:56:04.306602 12918 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:04.322182 12887 raft_consensus.cc:2781] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:56:04.323915 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.324164 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.324879 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:56:04.335690 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 184522us
I20240426 09:56:04.332495 12888 consensus_queue.cc:1046] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.004s
I20240426 09:56:04.336611 12887 raft_consensus.cc:686] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:04.337322 12888 raft_consensus.cc:3032] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:56:04.337531 12887 consensus_queue.cc:237] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.337849 12918 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.339851 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.335072 10008 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:04.260805 12871 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:04.341275 12871 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:04.341842 12903 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:56:04.325459 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 176019us
I20240426 09:56:04.345512 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:04.340006 12918 leader_election.cc:288] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.345923 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.348906 12918 consensus_queue.cc:1046] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:56:04.262025 12816 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.342537 12900 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.351665 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.352898 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:04.354002 9936 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:04.332803 12890 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.355635 12816 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:04.355917 12900 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.356611 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.361349 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:04.361887 10618 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:04.365186 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.367158 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.368820 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.331499 10619 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 10.
I20240426 09:56:04.342605 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.367825 10007 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 5.
I20240426 09:56:04.371677 9936 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 6.
I20240426 09:56:04.369338 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:04.372735 10014 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:04.373097 9904 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.367216 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.375797 10012 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
I20240426 09:56:04.376964 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.378947 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.382999 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.378443 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.384196 10618 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:04.370903 12888 raft_consensus.cc:729] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:04.385248 12883 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.386931 10084 raft_consensus.cc:1226] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:04.387580 10054 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:04.385728 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.388815 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.390573 9904 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:04.391187 10585 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:56:04.392015 10585 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:04.392313 12922 raft_consensus.cc:2781] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:04.392761 12922 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:04.393086 12922 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:04.394016 10014 raft_consensus.cc:2454] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 6.
I20240426 09:56:04.394927 12900 raft_consensus.cc:2726] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 09:56:04.396095 12915 raft_consensus.cc:2781] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:56:04.396482 12915 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:04.396808 12915 raft_consensus.cc:3037] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:04.397219 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.397640 12883 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.355931 12890 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.400156 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:04.394084 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.402056 10585 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:56:04.404825 10613 raft_consensus.cc:1226] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 6. Current term is 7. Ops: []
I20240426 09:56:04.391491 12894 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 09:56:04.408179 10089 raft_consensus.cc:1226] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:04.401288 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.410478 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:04.407277 10056 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:04.412400 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:04.412933 10081 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:04.408416 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.412757 12903 raft_consensus.cc:2781] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:56:04.424655 12903 raft_consensus.cc:686] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 6 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:56:04.425493 12903 consensus_queue.cc:237] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.414706 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:04.427181 10620 raft_consensus.cc:1226] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 9. Current term is 10. Ops: []
I20240426 09:56:04.385715 12888 consensus_queue.cc:260] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.415453 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:04.415719 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:04.416297 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:04.405804 12901 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.417855 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:04.429765 10087 raft_consensus.cc:2379] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 8: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:04.415732 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:04.401985 12918 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader election won for term 10
I20240426 09:56:04.418804 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:04.423789 10086 raft_consensus.cc:1226] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 5. Current term is 6. Ops: []
I20240426 09:56:04.431425 10083 raft_consensus.cc:2379] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 8: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:04.425472 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:04.432926 10056 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:56:04.421434 10081 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:04.441149 10089 raft_consensus.cc:1226] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 6. Current term is 7. Ops: []
I20240426 09:56:04.413491 12894 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.446858 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.448200 12921 raft_consensus.cc:2726] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 09:56:04.447273 12894 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Advancing to term 12
I20240426 09:56:04.445930 12890 consensus_queue.cc:1046] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20240426 09:56:04.460250 10056 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.455503 9935 raft_consensus.cc:1226] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 9. Current term is 10. Ops: []
I20240426 09:56:04.460747 12890 raft_consensus.cc:3032] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:56:04.449332 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.461028 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.454700 10615 raft_consensus.cc:1226] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:04.461764 9938 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:56:04.455152 9937 raft_consensus.cc:1226] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 6. Current term is 7. Ops: []
I20240426 09:56:04.462714 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.464104 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.464395 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.464778 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.458456 9835 catalog_manager.cc:5484] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 7 to 8, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 8 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:56:04.465809 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.466737 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.481712 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.461920 12859 raft_consensus.cc:2781] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:56:04.483929 10056 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:04.484299 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.483925 9938 raft_consensus.cc:1226] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 8. Current term is 9. Ops: []
I20240426 09:56:04.489478 10056 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.461146 12890 raft_consensus.cc:729] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:04.486469 10082 consensus_queue.cc:237] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:04.491452 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.496465 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.497844 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.485407 12816 raft_consensus.cc:2741] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20240426 09:56:04.485867 12888 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:04.502987 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.502077 12908 raft_consensus.cc:2726] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:56:04.503580 10012 raft_consensus.cc:1226] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 5. Current term is 6. Ops: []
I20240426 09:56:04.488556 12859 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:04.505981 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.509285 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.512295 10084 raft_consensus.cc:1226] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:04.512107 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.488700 12915 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.516546 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.520006 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.506762 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.510304 12859 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:04.512979 12816 raft_consensus.cc:2781] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:04.513933 10611 raft_consensus.cc:1226] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [term 6 LEADER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20240426 09:56:04.552487 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.515720 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.516693 12890 consensus_queue.cc:260] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:04.567629 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.605238 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.604318 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.605307 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.607065 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.791425 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.589637 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:56:04.667449 9962 debug-util.cc:398] Leaking SignalData structure 0x7b0800368220 after lost signal to thread 8761
I20240426 09:56:04.790845 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:56:04.795928 9962 debug-util.cc:398] Leaking SignalData structure 0x7b08003d4bc0 after lost signal to thread 9873
I20240426 09:56:04.790328 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.602957 9836 catalog_manager.cc:5484] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 5 to 6, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 6 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:56:04.797351 12892 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:04.797770 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.797866 12892 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.798883 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.799011 12893 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:04.799612 12893 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.800292 12892 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.801218 12893 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.586977 12895 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.801831 12895 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.803126 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
W20240426 09:56:04.803300 9962 debug-util.cc:398] Leaking SignalData structure 0x7b08003484a0 after lost signal to thread 9965
I20240426 09:56:04.614070 12894 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:04.806656 12895 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:04.808372 9962 debug-util.cc:398] Leaking SignalData structure 0x7b08003ce8c0 after lost signal to thread 10040
W20240426 09:56:04.809468 9962 debug-util.cc:398] Leaking SignalData structure 0x7b08002e2140 after lost signal to thread 10114
W20240426 09:56:04.809736 9962 debug-util.cc:398] Leaking SignalData structure 0x7b0800281260 after lost signal to thread 10115
W20240426 09:56:04.810809 9962 debug-util.cc:398] Leaking SignalData structure 0x7b08003de260 after lost signal to thread 10644
I20240426 09:56:04.811193 12894 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 12 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.790411 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.817379 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.819648 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.551381 12816 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:04.616907 12915 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:04.823551 10586 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:04.824468 10586 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.825537 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.826117 9932 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 9.
W20240426 09:56:04.827347 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 337907us
I20240426 09:56:04.828143 10056 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:04.829648 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.835929 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:04.830365 9941 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:56:04.837134 9905 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:04.823040 12816 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:04.840078 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.840651 9932 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:56:04.825281 12918 raft_consensus.cc:686] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 10 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:04.843058 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.843510 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:04.594935 12922 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.599289 12871 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.826514 12903 raft_consensus.cc:2726] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 09:56:04.792996 12859 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.846632 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.847407 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.849316 10620 raft_consensus.cc:1226] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 5. Current term is 6. Ops: []
I20240426 09:56:04.830888 12919 raft_consensus.cc:2781] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:56:04.826402 12879 consensus_queue.cc:1046] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20240426 09:56:04.854070 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:04.854089 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.857568 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:04.864625 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:04.842339 12894 raft_consensus.cc:2781] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:56:04.841185 12901 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:04.867074 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.871212 12879 raft_consensus.cc:3032] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:56:04.871721 12919 raft_consensus.cc:686] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 6 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:56:04.871825 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:04.893684 9938 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:04.872323 10615 raft_consensus.cc:1226] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 5. Current term is 6. Ops: []
I20240426 09:56:04.855230 12895 consensus_queue.cc:1046] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20240426 09:56:04.875656 10611 raft_consensus.cc:1226] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5. Ops: []
I20240426 09:56:04.876672 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.877347 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:04.878216 12890 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:04.879244 10583 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:04.879976 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:04.880422 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:56:04.901901 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 73828.8us
I20240426 09:56:04.880626 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:04.881182 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:04.891479 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:04.892522 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.900157 10087 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
W20240426 09:56:04.894685 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 49336.9us
I20240426 09:56:04.893532 12879 raft_consensus.cc:729] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:56:04.896638 9932 raft_consensus.cc:1226] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 12 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 11. Current term is 12. Ops: []
I20240426 09:56:04.896660 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.911855 9935 raft_consensus.cc:1226] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:04.916042 9941 raft_consensus.cc:1226] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 8. Current term is 9. Ops: []
I20240426 09:56:04.894932 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:04.918329 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.918543 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.895303 12894 raft_consensus.cc:686] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:04.854595 12914 consensus_queue.cc:1046] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
W20240426 09:56:04.923444 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 55405.3us
I20240426 09:56:04.920989 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.924892 10583 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
W20240426 09:56:04.927956 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 81258.3us
I20240426 09:56:04.911624 12889 raft_consensus.cc:2726] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:04.930044 10089 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:56:04.930604 10089 raft_consensus.cc:595] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:04.911806 12895 consensus_queue.cc:1225] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer 1fee3da01ebc463396397057fdb81a68 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20240426 09:56:04.937083 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.924015 12901 raft_consensus.cc:2726] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:56:04.918700 12879 consensus_queue.cc:260] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.949095 9938 raft_consensus.cc:2454] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:56:04.938725 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:04.952366 9935 raft_consensus.cc:1226] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 6. Current term is 7. Ops: []
I20240426 09:56:04.923754 12892 consensus_queue.cc:1046] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20240426 09:56:04.953927 9932 raft_consensus.cc:1226] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 5. Current term is 6. Ops: []
I20240426 09:56:04.953774 12894 consensus_queue.cc:237] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.954330 12892 raft_consensus.cc:3032] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:56:04.939136 12933 raft_consensus.cc:3032] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:56:04.947192 12937 consensus_queue.cc:1046] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20240426 09:56:04.954746 12892 raft_consensus.cc:729] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:04.925101 12918 consensus_queue.cc:237] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.933645 12917 consensus_queue.cc:1046] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20240426 09:56:04.950135 12919 consensus_queue.cc:237] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.939446 12859 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.964027 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.969900 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:04.969677 12942 raft_consensus.cc:3032] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:56:04.947754 12888 consensus_queue.cc:1046] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.032s
I20240426 09:56:04.985102 12942 raft_consensus.cc:729] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:04.960175 12943 raft_consensus.cc:2726] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 09:56:04.963105 12900 raft_consensus.cc:3032] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:56:04.962447 12911 raft_consensus.cc:3032] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:56:04.986172 12900 raft_consensus.cc:729] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:04.986433 12911 raft_consensus.cc:729] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:04.986812 12917 raft_consensus.cc:3032] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 LEADER]: Stepping down as leader of term 9
I20240426 09:56:04.985940 12942 consensus_queue.cc:260] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.987255 12911 consensus_queue.cc:260] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.987588 12930 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:56:04.988397 12942 raft_consensus.cc:3037] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:04.985951 12888 raft_consensus.cc:3032] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:56:04.968633 12871 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:04.989070 12888 raft_consensus.cc:729] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:04.968040 12922 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:04.989871 12888 consensus_queue.cc:260] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.990535 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:04.968394 12933 raft_consensus.cc:729] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:04.988641 12911 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:04.991972 12933 consensus_queue.cc:260] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:04.992975 12933 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:04.995875 12879 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:04.997098 10616 raft_consensus.cc:1226] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 6 LEADER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 5. Current term is 6. Ops: []
I20240426 09:56:04.997934 12888 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:04.982014 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:56:05.028178 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 96625.8us
I20240426 09:56:05.036325 10619 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 11 LEARNER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20240426 09:56:05.037737 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.987167 12917 raft_consensus.cc:729] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:05.041041 10583 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
W20240426 09:56:05.043278 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 94375.3us
I20240426 09:56:05.044667 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:04.987018 12900 consensus_queue.cc:260] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.050462 12900 raft_consensus.cc:3037] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:04.970463 12892 consensus_queue.cc:260] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.070037 12892 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:05.065462 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:05.070281 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:05.070621 10016 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:05.066222 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:05.066365 9938 raft_consensus.cc:1226] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:05.067068 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:05.072379 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:05.067402 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:05.075207 12859 raft_consensus.cc:979] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee: : Instructing follower 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:56:05.075682 12859 raft_consensus.cc:1067] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Signalling peer 826f03b2fb9c4b9483623b041a1bf80e to start an election
I20240426 09:56:05.075371 12917 consensus_queue.cc:260] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.067704 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:05.076386 12917 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:05.068217 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:05.068883 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:05.078366 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:05.081336 10583 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:05.082470 12879 raft_consensus.cc:2726] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 09:56:05.082887 12901 raft_consensus.cc:2781] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:05.084827 12901 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:05.085058 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:05.085211 12901 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Advancing to term 6
W20240426 09:56:05.089030 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 59046.2us
I20240426 09:56:05.092334 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:05.093116 12816 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.094902 9935 raft_consensus.cc:1261] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20240426 09:56:05.097010 9982 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.097496 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:05.098203 9936 raft_consensus.cc:2379] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:05.098430 12937 consensus_queue.cc:1046] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 09:56:05.099536 12937 raft_consensus.cc:2726] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:05.100081 12937 raft_consensus.cc:3032] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 9 LEADER]: Stepping down as leader of term 9
I20240426 09:56:05.100407 12937 raft_consensus.cc:729] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:05.101138 12937 consensus_queue.cc:260] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.100165 10082 raft_consensus.cc:1226] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 3. Current term is 4. Ops: []
I20240426 09:56:05.098131 9982 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:05.107746 12914 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:05.108098 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:05.108516 12903 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:05.108736 12910 consensus_queue.cc:1046] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:56:05.109097 12903 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.110255 9982 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.111454 9982 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.115298 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:05.108129 12914 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:05.116510 12914 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:05.109726 12910 raft_consensus.cc:3032] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:56:05.116987 12910 raft_consensus.cc:729] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:05.117954 12910 consensus_queue.cc:260] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.119094 12910 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:05.119815 12903 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:05.119995 12816 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:05.102587 12937 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:05.122581 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:05.123193 10013 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
W20240426 09:56:05.127148 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 67877.7us
W20240426 09:56:05.128006 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 220637us
I20240426 09:56:05.128132 10055 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.129657 12942 raft_consensus.cc:2726] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:56:05.130966 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:05.132072 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:05.130556 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:05.112167 12911 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:56:05.133857 12935 raft_consensus.cc:2781] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:05.134258 12935 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:05.134227 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:05.135535 12901 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.136533 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:05.137491 12901 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:05.137706 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:05.138994 10011 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 8.
I20240426 09:56:05.141301 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:05.135311 12935 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:05.130234 10586 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.135385 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:05.161142 12943 raft_consensus.cc:2726] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:56:05.130219 10055 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:05.151975 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:05.162721 10055 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.163473 12888 raft_consensus.cc:2781] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:05.163859 12888 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:05.163028 9835 catalog_manager.cc:5484] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 5 to 6, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 6 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:56:05.164268 12888 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:05.161971 10586 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.166431 12943 raft_consensus.cc:2726] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:56:05.163801 10055 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.169580 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:05.171262 10056 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.152297 10013 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 6.
I20240426 09:56:05.153046 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:05.173297 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:05.177230 10084 raft_consensus.cc:1226] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 10 LEADER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 9. Current term is 10. Ops: []
I20240426 09:56:05.177943 10585 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:56:05.178853 10619 raft_consensus.cc:1226] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 9. Current term is 10. Ops: []
I20240426 09:56:05.181685 12930 consensus_queue.cc:1046] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:56:05.182039 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:05.183770 12883 raft_consensus.cc:3032] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:56:05.184118 12883 raft_consensus.cc:729] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:05.184593 10612 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20240426 09:56:05.175292 12859 raft_consensus.cc:2726] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:56:05.184876 12883 consensus_queue.cc:260] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.185245 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:05.186173 12883 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
W20240426 09:56:05.190923 10054 consensus_peers.cc:485] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b76245317c24e129e7f16cdb4d99007. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:56:05.181588 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:05.180289 12917 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.012s
I20240426 09:56:05.175055 12949 raft_consensus.cc:2781] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:56:05.193538 12949 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:05.197854 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:05.198200 12859 consensus_queue.cc:1046] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 09:56:05.198599 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:05.199092 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:05.201365 12859 raft_consensus.cc:3032] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:56:05.201712 12859 raft_consensus.cc:729] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:05.162977 12816 raft_consensus.cc:2726] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:05.202481 12859 consensus_queue.cc:260] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.203471 9938 raft_consensus.cc:1261] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20240426 09:56:05.203837 9938 pending_rounds.cc:76] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9: Aborting all ops after (but not including) 6
I20240426 09:56:05.204062 9938 pending_rounds.cc:97] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20240426 09:56:05.204311 9938 raft_consensus.cc:2866] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:56:05.193893 12949 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:05.206324 10012 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:05.211576 12949 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.212018 10055 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:56:05.213521 12816 raft_consensus.cc:2781] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:56:05.214702 10055 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.216046 10055 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:05.218065 10011 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 10.
I20240426 09:56:05.193795 10014 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:05.203852 12859 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
W20240426 09:56:05.236689 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 125289us
I20240426 09:56:05.219532 9904 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:05.239059 9982 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.239970 9904 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:05.240262 9982 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.240335 10055 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.240060 9979 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:05.236909 12816 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:05.242368 12910 raft_consensus.cc:2726] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:56:05.242285 12871 raft_consensus.cc:2726] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:56:05.243865 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:05.244379 9982 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
W20240426 09:56:05.244853 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 66985.5us
I20240426 09:56:05.242534 12816 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:05.246979 12917 raft_consensus.cc:2726] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:05.244871 12864 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20240426 09:56:05.247447 12918 raft_consensus.cc:2781] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:05.247707 12937 raft_consensus.cc:2726] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:56:05.243055 12893 raft_consensus.cc:2781] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:56:05.249265 12950 raft_consensus.cc:2781] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:56:05.239344 12930 raft_consensus.cc:2726] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:56:05.224768 10614 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:56:05.250102 12917 consensus_queue.cc:1046] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:56:05.250952 9979 leader_election.cc:302] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:05.255252 10055 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.255461 9979 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.256717 10055 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.213765 12949 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:05.256767 9979 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.253779 10009 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 8.
I20240426 09:56:05.260365 12937 raft_consensus.cc:2726] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:05.261482 12921 raft_consensus.cc:2726] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 09:56:05.253647 12943 consensus_queue.cc:1035] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20240426 09:56:05.250888 12918 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:05.265595 12918 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:05.266490 10055 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:05.249225 12893 raft_consensus.cc:686] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:05.268500 12864 raft_consensus.cc:2726] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 09:56:05.269332 12921 raft_consensus.cc:2726] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:05.269080 12893 consensus_queue.cc:237] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.270069 12937 raft_consensus.cc:2726] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:05.270790 10055 leader_election.cc:302] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:05.271203 9979 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.227280 12888 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.241618 12883 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:05.273312 12911 raft_consensus.cc:3032] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:56:05.275250 9979 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.276340 10055 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.272943 12816 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.283130 10055 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:05.277495 9979 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
W20240426 09:56:05.285563 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 71762.6us
I20240426 09:56:05.277913 12883 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:05.284426 10054 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:05.179417 12894 consensus_queue.cc:1046] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.072s
I20240426 09:56:05.225006 12892 consensus_queue.cc:1046] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20240426 09:56:05.285835 9982 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.253139 12950 raft_consensus.cc:686] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 6 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:56:05.292096 10009 raft_consensus.cc:1261] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20240426 09:56:05.278575 12942 raft_consensus.cc:2726] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:05.259610 12910 raft_consensus.cc:2726] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:56:05.224063 12914 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.293206 10014 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422"
dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:34378
I20240426 09:56:05.312762 10014 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:56:05.313199 10014 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:05.286221 9906 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:05.278622 12911 raft_consensus.cc:729] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:05.319378 12911 consensus_queue.cc:260] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.320665 12911 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:05.288416 12864 raft_consensus.cc:2726] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:56:05.283871 12949 raft_consensus.cc:2726] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:56:05.278828 12890 raft_consensus.cc:2726] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 09:56:05.284036 12917 raft_consensus.cc:2726] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:05.310616 12816 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:05.288543 12883 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:05.286953 12937 consensus_queue.cc:1046] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20240426 09:56:05.326189 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:05.326396 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:05.290079 12921 raft_consensus.cc:2726] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 09:56:05.327471 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:05.289795 12930 raft_consensus.cc:2781] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:56:05.344096 12930 raft_consensus.cc:686] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:05.344991 12930 consensus_queue.cc:237] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.328804 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:05.329548 10089 raft_consensus.cc:1226] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 4. Current term is 5. Ops: []
I20240426 09:56:05.335431 10013 raft_consensus.cc:1226] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 8. Current term is 9. Ops: []
I20240426 09:56:05.347613 10583 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:05.337002 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:05.348601 12915 raft_consensus.cc:2726] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:56:05.351866 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:05.354435 12940 consensus_queue.cc:1046] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 09:56:05.355404 12940 raft_consensus.cc:3032] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:56:05.355734 12940 raft_consensus.cc:729] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:56:05.356495 12940 consensus_queue.cc:260] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.357096 12894 raft_consensus.cc:2726] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 09:56:05.357663 12940 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:05.360208 12888 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:05.338044 12895 raft_consensus.cc:3032] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:56:05.361202 12895 raft_consensus.cc:729] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:05.362006 12895 consensus_queue.cc:260] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.363245 12895 raft_consensus.cc:3037] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:05.330453 12942 consensus_queue.cc:1046] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20240426 09:56:05.293757 12935 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.369618 12914 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:05.370260 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 84013.9us
W20240426 09:56:05.299437 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 44771.2us
I20240426 09:56:05.339937 10082 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20240426 09:56:05.340400 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:05.358031 12955 raft_consensus.cc:2726] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:05.406019 10583 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.407166 12894 consensus_queue.cc:1046] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20240426 09:56:05.410282 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 124725us
I20240426 09:56:05.330577 12910 raft_consensus.cc:2726] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 09:56:05.420423 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 49499.3us
I20240426 09:56:05.408401 9904 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:05.423218 12894 raft_consensus.cc:3032] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:56:05.423601 12894 raft_consensus.cc:729] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:05.424301 12894 consensus_queue.cc:260] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.425525 12894 raft_consensus.cc:3037] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:56:05.428241 9982 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:05.428392 10088 raft_consensus.cc:2454] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:56:05.428802 9979 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.369627 12935 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:05.438374 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:05.437186 9835 catalog_manager.cc:5484] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 9 to 10, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 10 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:56:05.418498 12903 raft_consensus.cc:2726] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 09:56:05.471519 10007 raft_consensus.cc:1261] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 09:56:05.443522 10014 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.443713 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:05.444667 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:05.476229 10011 raft_consensus.cc:1261] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20240426 09:56:05.477185 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:56:05.480616 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 194455us
I20240426 09:56:05.480535 10008 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20240426 09:56:05.487018 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:56:05.487326 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 78388.6us
I20240426 09:56:05.488302 9904 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:05.497269 10086 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 6.
I20240426 09:56:05.439018 12816 consensus_queue.cc:1035] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
W20240426 09:56:05.507709 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 212917us
I20240426 09:56:05.442284 12933 consensus_queue.cc:1046] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20240426 09:56:05.500869 12883 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.512056 12951 consensus_queue.cc:1035] T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20240426 09:56:05.513506 12962 raft_consensus.cc:2726] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:56:05.514307 12883 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:05.360818 12950 consensus_queue.cc:237] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.498908 12900 raft_consensus.cc:3032] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:56:05.519043 10014 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:05.533993 12969 raft_consensus.cc:2726] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20240426 09:56:05.535461 12957 log.cc:924] Time spent T 51125fbde06b4a588fb6171dec56a605 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20240426 09:56:05.542757 12933 raft_consensus.cc:3032] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Stepping down as leader of term 8
I20240426 09:56:05.543133 12933 raft_consensus.cc:729] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:05.543908 12933 consensus_queue.cc:260] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.544968 12933 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:05.546761 10614 raft_consensus.cc:1261] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20240426 09:56:05.528245 12900 raft_consensus.cc:729] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:05.548303 12900 consensus_queue.cc:260] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.508024 12887 consensus_queue.cc:1035] T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
I20240426 09:56:05.510010 12917 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 09:56:05.565039 12917 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:05.565451 12883 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:05.561520 12816 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 09:56:05.563520 10614 raft_consensus.cc:1261] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 09:56:05.563844 12914 raft_consensus.cc:3032] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Stepping down as leader of term 8
W20240426 09:56:05.566684 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 138508us
I20240426 09:56:05.591886 9982 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:56:05.571962 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 143222us
I20240426 09:56:05.592988 9979 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
W20240426 09:56:05.594084 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 102741us
I20240426 09:56:05.597055 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:05.597806 10086 raft_consensus.cc:1261] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240426 09:56:05.600190 9982 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:05.597815 10055 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:05.605293 9979 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:05.565469 12917 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:05.606452 9979 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.617480 10012 raft_consensus.cc:1226] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 5. Current term is 6. Ops: []
I20240426 09:56:05.619788 9979 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.609279 12890 raft_consensus.cc:2726] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:05.621570 12903 consensus_queue.cc:1035] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20240426 09:56:05.630600 12910 raft_consensus.cc:2726] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 09:56:05.631134 9979 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:05.566006 12883 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.609573 12859 raft_consensus.cc:2726] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:56:05.609685 12911 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:05.591285 12914 raft_consensus.cc:729] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:05.632505 9979 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.621428 12958 raft_consensus.cc:2726] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 09:56:05.635936 9982 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:05.616475 12937 raft_consensus.cc:2726] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:05.621641 12959 raft_consensus.cc:2781] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 09:56:05.641633 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 153394us
I20240426 09:56:05.614835 12908 consensus_queue.cc:1035] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20240426 09:56:05.648983 10084 raft_consensus.cc:1226] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 5. Current term is 6. Ops: []
I20240426 09:56:05.637459 12890 raft_consensus.cc:2726] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:05.629475 12864 consensus_queue.cc:1046] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20240426 09:56:05.652927 9979 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.654866 9979 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
W20240426 09:56:05.641155 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 149178us
I20240426 09:56:05.639931 12911 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:05.659150 9938 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
mode: GRACEFUL
new_leader_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:46580
I20240426 09:56:05.659754 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:05.657471 10088 raft_consensus.cc:1226] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 5. Current term is 6. Ops: []
I20240426 09:56:05.635344 12942 consensus_queue.cc:1046] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.017s
I20240426 09:56:05.616693 12962 consensus_queue.cc:1046] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20240426 09:56:05.661456 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:05.643743 12959 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:05.662477 9938 raft_consensus.cc:595] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Received request to transfer leadership to TS d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:05.662709 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:05.663635 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:05.664175 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:05.666203 9979 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.665443 9906 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:05.667210 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:05.667443 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:05.668030 9934 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:05.670064 9936 raft_consensus.cc:1226] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 8. Current term is 9. Ops: []
I20240426 09:56:05.668514 9935 raft_consensus.cc:1226] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:05.653198 12937 raft_consensus.cc:2726] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:05.662078 12911 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:05.677675 9906 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:05.684685 9934 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:56:05.668486 12959 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:05.699410 9906 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:05.700553 9906 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:05.661001 12964 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20240426 09:56:05.693095 12937 raft_consensus.cc:2726] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 09:56:05.667543 12930 consensus_queue.cc:1035] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20240426 09:56:05.735522 10614 raft_consensus.cc:1226] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6. Ops: []
W20240426 09:56:05.736490 12975 log.cc:924] Time spent T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20240426 09:56:05.737382 12973 raft_consensus.cc:2726] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:05.737975 12967 raft_consensus.cc:2781] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:56:05.738966 12924 raft_consensus.cc:2726] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:56:05.739981 12965 raft_consensus.cc:2726] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 09:56:05.740382 12974 raft_consensus.cc:2726] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:05.748531 12883 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:05.749572 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:05.750236 10614 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:56:05.753607 10054 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:05.754509 12887 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:05.754925 12887 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:05.755251 12887 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:05.740136 12961 raft_consensus.cc:2726] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:56:05.768085 12887 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.770908 12887 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:05.771145 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:05.771837 10614 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:05.778257 10614 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 9.
I20240426 09:56:05.670646 12914 consensus_queue.cc:260] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.780315 12914 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:05.779384 10054 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:05.713827 12910 raft_consensus.cc:2726] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:56:05.783262 12887 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader election won for term 9
I20240426 09:56:05.786547 12950 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:05.787115 12950 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.784948 12887 raft_consensus.cc:686] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:05.790504 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:05.796665 10088 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 8.
I20240426 09:56:05.797694 10583 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:05.798491 12901 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:05.799839 12901 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:05.800207 12901 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:05.798980 12967 raft_consensus.cc:686] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:05.711139 12895 raft_consensus.cc:2726] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240426 09:56:05.815989 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 147043us
I20240426 09:56:05.811820 12967 consensus_queue.cc:237] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.812718 12955 raft_consensus.cc:3032] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:56:05.789631 12950 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:05.840631 10056 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.869673 10612 raft_consensus.cc:1261] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
W20240426 09:56:05.873446 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 207305us
I20240426 09:56:05.874205 9979 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:05.875123 12981 raft_consensus.cc:2726] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 09:56:05.875732 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 175260us
I20240426 09:56:05.876395 9979 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.882710 9934 raft_consensus.cc:1261] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240426 09:56:05.884757 12900 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:05.818202 12910 consensus_queue.cc:1046] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.069s
I20240426 09:56:05.887491 12910 raft_consensus.cc:3032] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:56:05.887903 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:05.888734 9934 raft_consensus.cc:2379] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 8: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:05.890095 12983 consensus_queue.cc:1035] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 09:56:05.890910 12879 consensus_queue.cc:1035] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:56:05.728315 12918 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.892861 12918 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:05.887514 12883 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:05.840343 12955 raft_consensus.cc:729] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:05.894271 12955 consensus_queue.cc:260] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.895476 12955 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:05.897476 12959 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.888038 12859 raft_consensus.cc:2726] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 09:56:05.893596 12883 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.887874 12972 consensus_queue.cc:1046] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.251s
I20240426 09:56:05.905002 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:56:05.875082 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 173436us
I20240426 09:56:05.898530 10016 raft_consensus.cc:1226] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 5. Current term is 6. Ops: []
I20240426 09:56:05.908325 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:05.915493 10008 raft_consensus.cc:2379] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 8: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:05.901460 9836 catalog_manager.cc:5484] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 reported cstate change: term changed from 6 to 7, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 7 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:56:05.921206 10056 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.932693 10088 raft_consensus.cc:1226] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:05.878060 9906 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:05.947053 10086 raft_consensus.cc:1226] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7. Ops: []
I20240426 09:56:05.935074 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:05.950663 9938 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:56:05.887840 12910 raft_consensus.cc:729] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:05.950704 10056 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.903752 12937 raft_consensus.cc:3032] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Stepping down as leader of term 7
W20240426 09:56:05.968358 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 90366us
I20240426 09:56:05.969254 9906 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:05.901975 12883 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:05.981348 9838 catalog_manager.cc:5484] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 reported cstate change: term changed from 5 to 6, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 1fee3da01ebc463396397057fdb81a68 (127.8.141.132). New cstate: current_term: 6 leader_uuid: "1fee3da01ebc463396397057fdb81a68" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 09:56:05.951783 8758 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240426 09:56:05.965556 10056 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.878060 9979 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.965582 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:05.983389 9932 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 9
W20240426 09:56:05.983537 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 105551us
I20240426 09:56:05.984239 8758 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 09:56:05.983846 12973 raft_consensus.cc:2781] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:05.984815 12973 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:05.985191 12973 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:05.985474 9979 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.986199 12961 raft_consensus.cc:2726] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:56:05.987172 12961 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 09:56:05.987704 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:56:05.966090 10082 raft_consensus.cc:1226] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7. Ops: []
I20240426 09:56:05.963924 9982 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:05.991386 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:05.992619 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:05.973613 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:05.975075 9940 raft_consensus.cc:1261] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20240426 09:56:05.994102 9938 raft_consensus.cc:2379] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 8: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:05.997059 12973 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.908545 12981 raft_consensus.cc:2726] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:05.894196 12890 raft_consensus.cc:3032] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 11 LEADER]: Stepping down as leader of term 11
I20240426 09:56:06.001729 12890 raft_consensus.cc:729] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
W20240426 09:56:05.999531 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 49901.2us
I20240426 09:56:06.004048 9906 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
W20240426 09:56:06.045913 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 80424.2us
I20240426 09:56:06.069635 10056 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:05.968029 12937 raft_consensus.cc:729] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
W20240426 09:56:06.063031 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 80509.2us
I20240426 09:56:06.062575 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
W20240426 09:56:06.047749 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 83937us
I20240426 09:56:05.952056 12894 raft_consensus.cc:2726] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:05.966722 12930 raft_consensus.cc:2726] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:56:05.906306 12962 raft_consensus.cc:3032] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Stepping down as leader of term 8
W20240426 09:56:06.068571 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 99254.8us
I20240426 09:56:05.902201 12859 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:05.903757 12917 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:05.898322 12901 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:06.077718 10086 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 9.
I20240426 09:56:05.944541 12871 consensus_queue.cc:1046] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.101s
I20240426 09:56:06.006732 12924 raft_consensus.cc:2726] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:06.079876 12888 raft_consensus.cc:2726] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 09:56:06.048974 12883 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:06.063127 12816 raft_consensus.cc:2726] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 09:56:06.072181 12972 raft_consensus.cc:2781] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:56:06.092393 9979 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
W20240426 09:56:06.092586 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 89390.6us
I20240426 09:56:06.084095 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:06.003965 12890 consensus_queue.cc:260] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.12, Last appended by leader: 12, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:05.951920 12887 consensus_queue.cc:237] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:06.113754 10015 raft_consensus.cc:1226] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20240426 09:56:06.119045 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:06.107245 9906 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:06.087906 12962 raft_consensus.cc:729] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:06.122206 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:06.093966 12893 raft_consensus.cc:2726] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:06.092016 12959 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:06.091037 12911 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:06.133905 9906 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:06.135974 9979 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:06.088198 12918 consensus_queue.cc:1046] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20240426 09:56:06.124694 12984 consensus_queue.cc:1046] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.155s
I20240426 09:56:06.119699 12955 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election won for term 9
I20240426 09:56:06.094409 12892 consensus_queue.cc:1046] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20240426 09:56:06.080169 12910 consensus_queue.cc:260] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:06.157269 9906 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:06.159390 10012 raft_consensus.cc:1226] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 8. Current term is 9. Ops: []
I20240426 09:56:06.161513 12901 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:06.105901 12883 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:06.163945 12883 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:06.165772 12991 ts_tablet_manager.cc:902] T 1b76245317c24e129e7f16cdb4d99007 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:06.166836 12979 log.cc:924] Time spent T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20240426 09:56:06.167017 10088 raft_consensus.cc:3032] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:56:06.167409 10088 raft_consensus.cc:729] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:06.168093 10088 consensus_queue.cc:260] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:06.168246 12937 consensus_queue.cc:260] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:06.169186 10088 raft_consensus.cc:3037] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:06.176491 12911 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:06.177415 12910 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:06.098562 12958 consensus_queue.cc:1046] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20240426 09:56:06.188856 9932 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 9.
I20240426 09:56:06.094558 12859 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:06.130823 12972 raft_consensus.cc:686] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:06.194818 10088 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:56:06.149137 12942 raft_consensus.cc:2726] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20240426 09:56:06.209189 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 50711.5us
I20240426 09:56:06.139678 12890 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Advancing to term 12
W20240426 09:56:06.228242 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 65250.3us
I20240426 09:56:06.185580 12970 raft_consensus.cc:2726] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:56:06.138114 12965 raft_consensus.cc:2726] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240426 09:56:06.231925 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 151000us
I20240426 09:56:06.233935 10088 raft_consensus.cc:1261] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:56:06.196856 12915 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:06.186663 12955 raft_consensus.cc:686] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:06.227840 12893 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:06.223876 12973 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:06.239614 10083 raft_consensus.cc:1226] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5. Ops: []
W20240426 09:56:06.241909 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 92257.8us
W20240426 09:56:06.185355 12983 consensus_queue.cc:1159] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f12521b6d51242629cf4297864df00b9" responder_term: 5 status { last_received { term: 5 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 5 index: 3 } }
I20240426 09:56:06.244717 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:06.245329 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:06.245409 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:06.186239 12889 consensus_queue.cc:1046] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20240426 09:56:06.249464 9941 raft_consensus.cc:2379] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 10: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
W20240426 09:56:06.250648 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 93884.7us
I20240426 09:56:06.251353 9906 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:06.252348 9906 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:06.253363 9906 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:06.254190 12963 raft_consensus.cc:2726] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:06.254801 12963 raft_consensus.cc:2726] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240426 09:56:06.255151 12982 log.cc:924] Time spent T ddc5aa33dc824544ade18161bf2800a6 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.183s user 0.001s sys 0.000s
I20240426 09:56:06.255484 12985 raft_consensus.cc:2726] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:56:06.255791 12894 raft_consensus.cc:2726] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 09:56:06.257555 9906 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:06.258666 9906 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:06.261021 9906 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:06.261902 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:06.288895 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:06.290294 10010 raft_consensus.cc:2379] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 10: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:06.255339 12963 raft_consensus.cc:2726] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:56:06.291630 9906 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:06.250353 12990 consensus_queue.cc:1046] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20240426 09:56:06.307689 9906 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:06.308626 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:06.262028 12970 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:06.239336 12915 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:06.311887 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 307815us
I20240426 09:56:06.310329 9906 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
W20240426 09:56:06.297415 13001 log.cc:924] Time spent T 51125fbde06b4a588fb6171dec56a605 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
W20240426 09:56:06.314563 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 65362.4us
I20240426 09:56:06.287878 12894 raft_consensus.cc:2726] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 09:56:06.316033 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:06.246161 12893 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:06.323253 9941 raft_consensus.cc:2379] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 8: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:06.227355 12962 consensus_queue.cc:260] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:06.310024 12971 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20240426 09:56:06.294838 12958 raft_consensus.cc:2726] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 09:56:06.327980 9906 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:06.301868 12917 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:06.332885 9906 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:06.334626 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:06.335188 10012 raft_consensus.cc:2362] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 8. Re-sending same reply.
I20240426 09:56:06.305228 12965 raft_consensus.cc:2726] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 09:56:06.310689 12985 raft_consensus.cc:2726] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 09:56:06.342365 10016 raft_consensus.cc:1226] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 5. Current term is 6. Ops: []
I20240426 09:56:06.343184 10012 raft_consensus.cc:1261] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20240426 09:56:06.345187 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 56456.1us
I20240426 09:56:06.345811 9941 raft_consensus.cc:1261] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:56:06.330338 12964 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 09:56:06.347119 9979 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:06.348846 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:06.353412 9979 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:06.351629 13008 consensus_queue.cc:1046] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20240426 09:56:06.355772 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:06.342185 12900 raft_consensus.cc:2726] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:06.358692 12893 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:06.360085 12915 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:06.310508 12972 consensus_queue.cc:237] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:06.313640 12970 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:06.345568 12990 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:56:06.298110 12955 consensus_queue.cc:237] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:06.362741 12997 log.cc:924] Time spent T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20240426 09:56:06.383213 13008 raft_consensus.cc:3032] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Stepping down as leader of term 8
I20240426 09:56:06.383634 13008 raft_consensus.cc:729] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:06.364516 12986 raft_consensus.cc:3032] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:56:06.384279 12986 raft_consensus.cc:729] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:06.384370 13008 consensus_queue.cc:260] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:06.385676 13008 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:06.387005 12969 raft_consensus.cc:979] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9: : Instructing follower d3a480e831d649139ff016fb68aee3ee to start an election
W20240426 09:56:06.366983 13010 log.cc:924] Time spent T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20240426 09:56:06.387684 12816 raft_consensus.cc:3032] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:56:06.364791 12968 raft_consensus.cc:2726] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 09:56:06.388072 12816 raft_consensus.cc:729] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:06.391599 12816 consensus_queue.cc:260] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:06.392302 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 43557us
I20240426 09:56:06.392704 12816 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:06.367027 12970 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:06.385075 12986 consensus_queue.cc:260] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:06.395570 12986 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:06.396317 12937 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
W20240426 09:56:06.366539 12999 log.cc:924] Time spent T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
I20240426 09:56:06.403384 12976 raft_consensus.cc:3032] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:56:06.403785 12976 raft_consensus.cc:729] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:06.375460 12859 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:56:06.404186 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 51421.8us
I20240426 09:56:06.404558 12976 consensus_queue.cc:260] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:06.406033 12964 raft_consensus.cc:686] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [term 10 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:06.406309 12990 raft_consensus.cc:686] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 8 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:06.406503 12895 raft_consensus.cc:3032] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:56:06.406930 12895 raft_consensus.cc:729] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:06.407182 12962 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:06.407711 12969 raft_consensus.cc:1067] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Signalling peer d3a480e831d649139ff016fb68aee3ee to start an election
W20240426 09:56:06.382105 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 66828.8us
I20240426 09:56:06.408754 10056 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:06.409868 10056 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:06.410542 12970 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:06.382261 12933 consensus_queue.cc:1035] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20240426 09:56:06.411432 12892 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 09:56:06.412084 12933 raft_consensus.cc:2726] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 09:56:06.376024 12935 raft_consensus.cc:3032] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Stepping down as leader of term 9
I20240426 09:56:06.413137 12935 raft_consensus.cc:729] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:06.413690 12892 consensus_queue.cc:1046] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:56:06.413931 12935 consensus_queue.cc:260] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:06.414791 12968 raft_consensus.cc:3032] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:56:06.415138 12968 raft_consensus.cc:729] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:06.415365 10056 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:06.407670 12895 consensus_queue.cc:260] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:06.415985 12968 consensus_queue.cc:260] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:06.417156 12958 consensus_queue.cc:1159] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "826f03b2fb9c4b9483623b041a1bf80e" responder_term: 5 status { last_received { term: 5 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 5 index: 4 } }
I20240426 09:56:06.417347 12976 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:06.416970 12895 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:06.419315 12964 consensus_queue.cc:237] T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:06.423967 10089 raft_consensus.cc:1261] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20240426 09:56:06.425259 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 74086.4us
I20240426 09:56:06.428138 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:06.428772 10016 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 7.
I20240426 09:56:06.430223 10012 raft_consensus.cc:1261] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240426 09:56:06.372397 10083 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 8.
W20240426 09:56:06.441367 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 88005.3us
I20240426 09:56:06.447403 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:06.450845 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:06.451594 9940 raft_consensus.cc:2379] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 9: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:06.454716 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:06.456992 9938 raft_consensus.cc:2379] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 12: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:06.415273 12935 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:06.459161 12911 consensus_queue.cc:1035] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 09:56:06.461091 10585 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:56:06.415663 12859 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:06.418732 12983 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:06.459913 12917 raft_consensus.cc:2726] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 09:56:06.477993 12968 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:06.466310 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:06.479758 12963 raft_consensus.cc:3032] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:56:06.480149 12963 raft_consensus.cc:729] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:06.483323 12903 raft_consensus.cc:2781] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:06.467813 10583 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:06.466379 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:06.489393 10013 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 9.
I20240426 09:56:06.490456 10055 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:06.452255 12910 raft_consensus.cc:2726] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:56:06.491783 9979 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
W20240426 09:56:06.509270 9935 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.004s sys 0.033s
W20240426 09:56:06.509670 9935 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.529s user 0.004s sys 0.033s
I20240426 09:56:06.514328 10083 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:59762
I20240426 09:56:06.515003 10083 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 09:56:06.515390 10083 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:06.470170 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:06.516950 10617 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:56:06.518080 9905 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:06.519901 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:06.472527 12970 raft_consensus.cc:2726] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 09:56:06.520593 10013 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:56:06.534155 10089 raft_consensus.cc:1226] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 6. Current term is 7. Ops: []
I20240426 09:56:06.510170 12930 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:06.475438 12900 consensus_queue.cc:1046] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20240426 09:56:06.525843 12871 raft_consensus.cc:2781] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:06.541393 12871 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:06.541728 13014 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:56:06.542264 13014 raft_consensus.cc:686] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 8 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:56:06.474174 12950 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20240426 09:56:06.486488 12859 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:06.471110 12969 consensus_queue.cc:1035] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20240426 09:56:06.556634 12903 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:06.536233 12962 raft_consensus.cc:2726] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 09:56:06.553778 10083 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:06.543128 13014 consensus_queue.cc:237] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:06.541774 12871 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:06.569741 10083 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:06.568078 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:06.407223 12990 consensus_queue.cc:237] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:06.588861 12987 raft_consensus.cc:3032] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:56:06.589275 12987 raft_consensus.cc:729] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:06.590039 12987 consensus_queue.cc:260] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:06.581091 9938 raft_consensus.cc:3032] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:56:06.585173 13003 consensus_queue.cc:1035] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.019s
I20240426 09:56:06.591466 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:06.652511 9938 raft_consensus.cc:729] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:06.652920 10013 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:06.594916 12963 consensus_queue.cc:260] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:06.540033 12930 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:06.603159 10011 raft_consensus.cc:2396] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for term 12 because replica has last-logged OpId of term: 11 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20240426 09:56:06.579725 12917 consensus_queue.cc:1046] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
W20240426 09:56:06.613162 13015 log.cc:924] Time spent T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
W20240426 09:56:06.628654 13018 log.cc:924] Time spent T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20240426 09:56:06.640075 12859 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:56:06.641304 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 48246.3us
W20240426 09:56:06.641795 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 47026.8us
W20240426 09:56:06.645820 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 45773.2us
I20240426 09:56:06.647184 9835 catalog_manager.cc:5484] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 4 to 5, leader changed from 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 5 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:56:06.664615 10089 raft_consensus.cc:1261] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:56:06.664469 9835 catalog_manager.cc:5484] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 09:56:06.653270 9938 consensus_queue.cc:260] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:06.667963 9939 raft_consensus.cc:1261] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240426 09:56:06.565716 12903 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Advancing to term 8
W20240426 09:56:06.657768 13021 log.cc:924] Time spent T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.073s user 0.002s sys 0.000s
I20240426 09:56:06.667160 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:06.672350 10055 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:06.670920 9938 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:06.677279 10013 raft_consensus.cc:2454] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:06.678222 10055 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:06.661799 12930 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:06.715260 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:06.803262 10015 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:56:06.858302 9938 raft_consensus.cc:2454] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
W20240426 09:56:06.803139 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 124993us
I20240426 09:56:06.864174 10055 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:06.710886 13008 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Advancing to term 12
I20240426 09:56:06.868716 12918 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:06.869879 12949 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:56:06.879860 13017 log.cc:924] Time spent T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.228s user 0.000s sys 0.000s
I20240426 09:56:06.802755 12987 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:06.802722 12951 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:06.860585 12889 raft_consensus.cc:2781] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:56:06.851233 12937 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:06.683245 12908 consensus_queue.cc:1035] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.185s
I20240426 09:56:06.803073 12963 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:06.707782 12890 consensus_queue.cc:1035] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.159s
I20240426 09:56:06.924837 10084 raft_consensus.cc:1261] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 09:56:06.883924 10617 raft_consensus.cc:1261] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240426 09:56:06.930832 10013 raft_consensus.cc:1261] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20240426 09:56:06.934998 13027 log.cc:924] Time spent T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20240426 09:56:06.880314 12915 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:06.921025 12889 raft_consensus.cc:686] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:06.964658 9934 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:06.891592 12918 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:06.994071 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 2342) took 1049 ms. Trace:
I20240426 09:56:06.994513 9935 rpcz_store.cc:276] 0426 09:56:05.944452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:05.947254 (+ 2802us) service_pool.cc:225] Handling call
0426 09:56:05.947694 (+ 440us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:05.948013 (+ 319us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:05.948036 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:05.952790 (+ 4754us) log.cc:842] Serialized 506 byte log entry
0426 09:56:06.510018 (+557228us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:56:06.510062 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:06.510335 (+ 273us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:06.879193 (+368858us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:56:06.982641 (+103448us) raft_consensus.cc:1671] finished
0426 09:56:06.982663 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:06.994003 (+ 11340us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39779200,"thread_start_us":528967,"threads_started":1,"wal-append.queue_time_us":698790}
I20240426 09:56:06.912678 12951 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:06.918574 12937 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.006346 9935 raft_consensus.cc:1261] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:56:06.961828 13023 raft_consensus.cc:2781] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:07.012311 9934 raft_consensus.cc:1261] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 09:56:06.949236 12914 consensus_queue.cc:1035] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
W20240426 09:56:06.992590 12983 consensus_meta.cc:287] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.505s user 0.001s sys 0.007s
I20240426 09:56:06.895830 12949 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:07.079878 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 54692.9us
I20240426 09:56:06.955049 12895 consensus_queue.cc:1035] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20240426 09:56:06.950335 12915 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:07.096076 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 74373.2us
I20240426 09:56:07.022353 12990 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.095750 12983 raft_consensus.cc:2726] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 09:56:06.998430 12933 consensus_queue.cc:1035] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.174s
I20240426 09:56:07.196659 13007 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.197355 13007 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.075345 12962 consensus_queue.cc:1035] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.114s
I20240426 09:56:07.168691 12900 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.091773 12991 tablet_copy_client.cc:308] T 1b76245317c24e129e7f16cdb4d99007 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Beginning tablet copy session from remote peer at address 127.8.141.131:34821
I20240426 09:56:07.184850 12911 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:07.304744 12900 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.322247 12933 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.183275 12984 consensus_queue.cc:1035] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.127s
I20240426 09:56:07.218937 12962 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:56:07.310825 13035 log.cc:924] Time spent T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.290s user 0.000s sys 0.001s
I20240426 09:56:07.322569 12911 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.330235 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:07.309291 13005 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.185868 12990 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:07.329500 12935 consensus_meta.cc:287] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.857s user 0.003s sys 0.006s
I20240426 09:56:07.338966 10084 raft_consensus.cc:2454] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:56:07.346316 9904 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:07.329716 12967 raft_consensus.cc:3032] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Stepping down as leader of term 6
W20240426 09:56:07.322525 12871 consensus_meta.cc:287] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.686s user 0.007s sys 0.001s
I20240426 09:56:07.322484 12900 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:07.379890 13023 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:07.380419 13023 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:07.200937 12889 consensus_queue.cc:237] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.388473 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:07.389545 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.389128 10012 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:07.399624 10055 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:07.357288 12968 raft_consensus.cc:2781] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:56:07.352476 12967 raft_consensus.cc:729] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:07.337069 12933 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.337729 12962 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.414239 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:07.348455 13005 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.408085 12921 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.411219 12968 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.410189 12918 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:07.411324 13016 raft_consensus.cc:2781] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:07.431458 12937 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:07.434389 12915 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:07.444639 13037 log.cc:924] Time spent T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.090s user 0.001s sys 0.000s
I20240426 09:56:07.417169 12984 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:56:07.441838 13008 consensus_meta.cc:287] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.560s user 0.006s sys 0.002s
I20240426 09:56:07.442348 12967 consensus_queue.cc:260] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.443301 12888 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.445688 12968 raft_consensus.cc:3037] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
W20240426 09:56:07.443538 12903 consensus_meta.cc:287] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68: Time spent flushing consensus metadata: real 0.760s user 0.008s sys 0.000s
I20240426 09:56:07.444469 12962 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:07.432361 13006 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:07.445394 10016 raft_consensus.cc:1261] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240426 09:56:07.420656 13007 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:07.420782 12951 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:07.448493 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.363075 12871 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.489624 12871 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:07.452921 12987 consensus_meta.cc:287] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.557s user 0.004s sys 0.004s
I20240426 09:56:07.453823 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.427659 13043 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.491070 13043 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.492251 13054 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:07.491961 12990 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:07.478444 12903 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.493106 13043 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:07.459789 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.431825 10617 raft_consensus.cc:2454] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
W20240426 09:56:07.452632 12963 consensus_meta.cc:287] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.527s user 0.002s sys 0.006s
I20240426 09:56:07.461023 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.445870 12900 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.446416 12949 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:07.477677 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.446893 13016 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:07.480108 12937 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.480585 12933 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:07.480993 13005 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:07.477576 12888 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.443655 13023 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.483651 12911 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:07.495790 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
W20240426 09:56:07.490978 13038 log.cc:924] Time spent T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20240426 09:56:07.515583 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:07.490558 9934 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:56:07.516273 10616 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:56:07.516767 12888 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:07.506035 13006 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.510195 12937 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.518582 13023 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:56:07.520275 13028 log.cc:924] Time spent T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.872s user 0.007s sys 0.005s
I20240426 09:56:07.504225 9937 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 6.
I20240426 09:56:07.444689 12921 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.520778 9905 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:07.521275 12951 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.507465 13054 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:07.521770 12951 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.523061 13054 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:07.523661 12951 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:07.524355 13053 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.524839 13053 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.506671 13016 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:07.461027 12984 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.530311 10086 raft_consensus.cc:1261] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:56:07.493036 12990 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.475380 13008 raft_consensus.cc:2726] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 09:56:07.514608 10612 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Advancing to term 7
W20240426 09:56:07.532310 13032 log.cc:924] Time spent T 9c2caa6ec04942588af2288b0ac9ef6b P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.181s user 0.000s sys 0.000s
I20240426 09:56:07.527740 12903 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:07.529064 10616 raft_consensus.cc:1226] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:07.551795 12984 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:07.552588 13025 log.cc:924] Time spent T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.902s user 0.001s sys 0.011s
I20240426 09:56:07.552364 12969 consensus_queue.cc:1046] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 09:56:07.518697 13006 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:07.548897 12921 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:07.532436 12895 raft_consensus.cc:2781] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20240426 09:56:07.556301 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 145019us
I20240426 09:56:07.526744 13053 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:07.554695 12930 consensus_meta.cc:287] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.704s user 0.006s sys 0.003s
I20240426 09:56:07.561686 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.562445 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.563128 9934 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:56:07.564882 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.566344 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:07.518921 12937 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:07.508348 12900 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.562263 9937 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:07.572085 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.579784 9935 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:56:07.562486 10612 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
W20240426 09:56:07.578835 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 160463us
I20240426 09:56:07.581015 9905 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:07.592382 10056 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:07.595436 10086 raft_consensus.cc:1261] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240426 09:56:07.601976 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:07.602643 10086 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:56:07.546690 12990 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.606081 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:07.607887 10086 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 10.
I20240426 09:56:07.595597 12921 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.609889 10056 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:07.596674 13006 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.595204 12963 raft_consensus.cc:2781] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:56:07.614500 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:07.640578 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:07.641003 10011 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 7
I20240426 09:56:07.614962 10586 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:07.615823 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:07.642210 10014 raft_consensus.cc:2379] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 10: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:56:07.624979 9935 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 09:56:07.645045 9932 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:56:07.638753 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:07.649397 9904 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:07.650059 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:07.651437 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:07.652038 10612 raft_consensus.cc:2454] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 3.
I20240426 09:56:07.650671 10015 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:56:07.657613 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:07.658242 10013 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:56:07.643096 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:07.658895 10620 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:56:07.660079 13051 raft_consensus.cc:2781] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:07.660475 13051 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:07.660792 13051 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:07.658300 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:07.595072 12937 consensus_queue.cc:1035] T 3c3eaa7da54b489e99a7451385057422 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20240426 09:56:07.665892 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.665901 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.667484 10611 raft_consensus.cc:2379] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 6: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:56:07.609366 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:07.607784 13060 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:07.643409 13006 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.699697 13060 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.701329 13064 raft_consensus.cc:2781] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:07.701903 13064 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.702502 13064 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:07.681399 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:07.590822 12930 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.680562 9937 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
W20240426 09:56:07.689069 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 79240.3us
I20240426 09:56:07.705284 10056 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:07.610476 12950 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.640436 12816 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:07.697590 13069 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 09:56:07.701409 13006 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:07.713634 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:07.714864 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:07.715451 10014 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:56:07.677628 9982 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:56:07.706823 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 91934.8us
W20240426 09:56:07.707515 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 91437us
W20240426 09:56:07.708344 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 62425.5us
I20240426 09:56:07.711221 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:07.704535 12930 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:07.713169 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:07.663859 12963 raft_consensus.cc:686] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:07.719671 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:07.719637 10086 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:56:07.720371 10090 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:56:07.721562 13061 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:07.722044 13061 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.722383 13061 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:07.724048 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:07.724493 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:07.725044 10082 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:56:07.704615 12990 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:07.639783 12921 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.726900 12900 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:07.662921 12893 consensus_queue.cc:1035] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20240426 09:56:07.713266 13069 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:07.728941 13069 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:07.729898 10054 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:07.730746 10054 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:07.730511 12968 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.731614 13062 raft_consensus.cc:2781] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:56:07.731985 13062 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.732336 13062 raft_consensus.cc:3037] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:07.732908 9938 raft_consensus.cc:2379] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:07.733062 12971 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.733521 12971 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.709748 10617 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:56:07.735432 12971 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:07.713019 12950 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.736725 10084 raft_consensus.cc:1226] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:07.737210 12950 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:07.710057 12816 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.738112 12816 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:07.727115 12889 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:07.740618 9979 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:07.749261 9979 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:07.750949 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.754222 12890 raft_consensus.cc:2781] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:07.754416 12937 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:07.720474 12963 consensus_queue.cc:237] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.745898 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.755932 12921 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:07.748507 12892 consensus_queue.cc:1046] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.008s
I20240426 09:56:07.732107 13059 raft_consensus.cc:2781] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:07.754659 12890 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.748337 12908 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.756840 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:07.780346 13055 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.004s
I20240426 09:56:07.780993 10015 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:56:07.780356 12908 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.761346 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.783169 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:07.783620 12908 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:07.761670 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:07.783805 10611 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:56:07.784545 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:07.766877 12937 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.756934 12889 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.785595 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:07.785821 12864 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:56:07.741679 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 64332.9us
I20240426 09:56:07.756785 12895 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.787128 10100 tablet_copy_service.cc:140] P d3a480e831d649139ff016fb68aee3ee: Received BeginTabletCopySession request for tablet 1b76245317c24e129e7f16cdb4d99007 from peer 1fee3da01ebc463396397057fdb81a68 ({username='slave'} at 127.0.0.1:48202)
I20240426 09:56:07.787685 10100 tablet_copy_service.cc:161] P d3a480e831d649139ff016fb68aee3ee: Beginning new tablet copy session on tablet 1b76245317c24e129e7f16cdb4d99007 from peer 1fee3da01ebc463396397057fdb81a68 at {username='slave'} at 127.0.0.1:48202: session id = 1fee3da01ebc463396397057fdb81a68-1b76245317c24e129e7f16cdb4d99007
I20240426 09:56:07.788283 9905 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:07.789654 13016 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.776790 13059 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.787067 12895 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:07.791209 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.792297 9935 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 9.
I20240426 09:56:07.764221 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:07.785504 12889 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:07.776603 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:07.796074 10619 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:56:07.764444 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:07.766460 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:07.767225 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:07.798624 10014 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
W20240426 09:56:07.764549 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 148014us
I20240426 09:56:07.771440 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:07.732503 12968 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:07.709180 13060 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:07.775548 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:07.804417 10586 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:07.777448 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:07.778842 12971 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.791752 12890 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:07.798058 12900 raft_consensus.cc:2781] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:07.798202 10612 raft_consensus.cc:3037] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Advancing to term 5
I20240426 09:56:07.762609 10011 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:07.798570 10090 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 8.
I20240426 09:56:07.818190 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:56:07.818560 13034 log.cc:924] Time spent T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.646s user 0.008s sys 0.000s
I20240426 09:56:07.819842 13051 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.801338 9980 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:56:07.801791 10583 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:07.823235 10615 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:07.823438 10089 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 579) took 1153 ms. Trace:
I20240426 09:56:07.823793 10089 rpcz_store.cc:276] 0426 09:56:06.670157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:06.670307 (+ 150us) service_pool.cc:225] Handling call
0426 09:56:06.670691 (+ 384us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:06.671004 (+ 313us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:06.671029 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:06.671405 (+ 376us) log.cc:842] Serialized 26 byte log entry
0426 09:56:06.883246 (+211841us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:56:06.883284 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:06.883351 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:07.817992 (+934641us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:56:07.823150 (+ 5158us) raft_consensus.cc:1671] finished
0426 09:56:07.823170 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:07.823346 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":137756,"raft.queue_time_us":90,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":36944,"spinlock_wait_cycles":6435968,"thread_start_us":211568,"threads_started":1,"wal-append.queue_time_us":317787}
I20240426 09:56:07.825698 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:07.826309 10089 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 6.
I20240426 09:56:07.805593 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.785430 12937 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:07.809470 10007 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 6.
I20240426 09:56:07.786283 12864 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.791620 13059 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:07.831580 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:07.840680 10583 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:07.841444 9904 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:56:07.832916 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:07.828495 9935 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:56:07.827965 9982 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:07.836474 10612 raft_consensus.cc:2454] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:56:07.837975 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:07.844089 9905 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:07.844851 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:07.845021 9982 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:07.810483 12935 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.827656 12900 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.817257 12908 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.849830 10056 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:07.853356 12816 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.853746 13065 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:07.854177 13065 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.854589 13065 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:07.858335 12864 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:07.858935 10085 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 5.
I20240426 09:56:07.851898 12921 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:07.859467 13075 raft_consensus.cc:2726] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:56:07.860347 10088 raft_consensus.cc:2379] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 9: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:07.861106 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:07.861703 10615 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:56:07.852298 13055 raft_consensus.cc:2781] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:07.862253 13055 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.862571 13055 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:07.850347 13060 raft_consensus.cc:2781] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:56:07.865041 10054 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:07.865531 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:07.866137 10011 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
I20240426 09:56:07.839831 13051 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:07.873364 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:07.871116 10055 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:07.871775 10585 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:56:07.867161 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:07.874914 9935 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:07.855746 12816 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:07.852003 12981 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:07.867475 12908 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.879561 12981 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.869980 12921 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.870244 12864 raft_consensus.cc:2781] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:07.882066 12864 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.882400 12864 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:07.882694 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:07.852248 12935 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.872829 13006 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:07.853991 13016 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:07.886108 9933 raft_consensus.cc:2379] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 9: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:07.877068 13053 raft_consensus.cc:2726] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 09:56:07.864518 12900 raft_consensus.cc:3037] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:07.918944 13068 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:07.816135 12971 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.863494 13060 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:07.938876 13060 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:07.939667 13067 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.904145 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:07.906752 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.908259 13006 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.943532 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.917569 12895 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.918538 12981 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:07.920764 12962 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
W20240426 09:56:07.923552 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 114818us
I20240426 09:56:07.924214 12950 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20240426 09:56:07.925586 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 53884.8us
I20240426 09:56:07.850207 12968 raft_consensus.cc:2781] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:56:07.930686 10081 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 09:56:07.931430 13016 raft_consensus.cc:2726] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20240426 09:56:07.932049 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 61002.2us
I20240426 09:56:07.918485 13046 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:07.932539 12915 raft_consensus.cc:2781] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:07.933908 12971 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:07.934633 13068 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.957597 13068 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:07.957259 13046 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.957279 12915 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.959278 12915 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:07.943248 12965 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.951005 12935 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:07.952932 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.937669 12889 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.960490 12965 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.961220 9941 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:56:07.963390 10056 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:07.964200 12889 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:07.964473 12971 raft_consensus.cc:2781] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:07.965297 12971 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.965448 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:07.966008 9941 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:07.966236 13046 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:07.967049 12965 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:07.968313 13046 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.968780 13046 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.970561 13046 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:07.971907 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.972265 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:07.972509 9940 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:07.972939 9936 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:56:07.973537 12971 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:07.972926 12889 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:07.976692 12889 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.973277 9933 raft_consensus.cc:2454] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:07.978231 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:07.979209 10081 raft_consensus.cc:2454] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:56:07.979979 12889 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:07.974359 10056 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:07.982699 10056 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:07.983290 12889 raft_consensus.cc:2781] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:07.984161 12889 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:07.984617 12889 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:07.986352 13016 raft_consensus.cc:2781] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:07.986793 13016 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:07.987120 13016 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:07.998066 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:07.998669 10015 raft_consensus.cc:2454] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
W20240426 09:56:07.999356 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 72617.8us
I20240426 09:56:08.004004 9934 raft_consensus.cc:2354] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 6. Current term is 7.
I20240426 09:56:08.006978 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:08.007676 12871 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:08.008162 12871 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:08.010082 12871 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:07.991181 12908 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:07.951803 12892 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:08.006898 9904 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:08.011778 12892 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:08.020695 13073 log.cc:924] Time spent T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.170s user 0.000s sys 0.000s
I20240426 09:56:08.021862 12892 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:08.024827 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 68894.1us
I20240426 09:56:08.025970 10585 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:08.027254 9940 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:56:07.918015 13064 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:07.995772 12990 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:08.039913 9935 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 9.
I20240426 09:56:08.010314 13065 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:07.995965 12930 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:08.022658 10100 tablet_copy_source_session.cc:205] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 09:56:08.041676 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:56:08.079672 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 215524us
I20240426 09:56:07.999168 13050 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:08.085537 10056 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:08.024888 12965 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:08.033947 12969 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:08.095306 12990 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:08.104995 12921 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:08.104053 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:08.022949 12968 raft_consensus.cc:686] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:07.996727 12962 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:08.113039 12816 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader election won for term 9
I20240426 09:56:08.121748 12816 raft_consensus.cc:686] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:08.113611 10082 raft_consensus.cc:2354] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 6. Current term is 7.
I20240426 09:56:08.041796 12951 raft_consensus.cc:2726] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 09:56:08.123653 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:08.128041 10081 raft_consensus.cc:2379] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 6: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:08.133044 9941 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 9.
I20240426 09:56:08.133641 13048 raft_consensus.cc:2781] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:08.135324 12816 consensus_queue.cc:237] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:08.149360 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:07.940248 13067 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:08.136987 12895 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:08.138419 12990 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:08.141099 13065 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:08.143270 12962 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:08.170852 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:08.171521 10619 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:08.172341 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:08.172987 10081 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 9.
I20240426 09:56:08.174316 10015 raft_consensus.cc:2396] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for term 13 because replica has last-logged OpId of term: 11 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20240426 09:56:08.153005 13048 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:08.023568 13062 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:08.136159 12989 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:08.159057 12943 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:08.123317 12965 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:08.156154 12991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b76245317c24e129e7f16cdb4d99007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:56:08.160313 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:08.186632 10084 raft_consensus.cc:2396] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for term 13 because replica has last-logged OpId of term: 11 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20240426 09:56:08.160079 12968 consensus_queue.cc:237] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:08.083894 12871 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:08.178088 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:08.110205 12930 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:08.154446 13072 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:08.151634 13055 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:08.183734 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 152689us
I20240426 09:56:08.134166 12890 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:08.161181 12973 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:08.116856 13050 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:08.193681 10087 raft_consensus.cc:2379] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 5: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:08.179603 13048 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:08.184139 12990 raft_consensus.cc:3032] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:56:08.198796 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:08.209260 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:08.209574 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:08.210222 10009 raft_consensus.cc:2354] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 8. Current term is 9.
W20240426 09:56:08.213675 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 127584us
I20240426 09:56:08.213824 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:08.211126 12890 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:08.156181 13013 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
I20240426 09:56:08.174881 13069 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:08.230960 10014 raft_consensus.cc:2379] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 9: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:56:08.236222 10619 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:08.227345 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:08.221611 10012 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 9.
I20240426 09:56:08.223863 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:08.237835 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:08.233422 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:08.233551 10612 raft_consensus.cc:2354] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 6. Current term is 7.
I20240426 09:56:08.227742 10056 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:08.240090 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:08.240689 9938 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:08.241118 9935 raft_consensus.cc:1261] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:56:08.242439 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:08.247680 9938 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 9.
I20240426 09:56:08.250109 10620 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 8.
I20240426 09:56:08.251281 12816 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader election won for term 9
I20240426 09:56:08.251772 12816 raft_consensus.cc:686] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:08.252490 12816 consensus_queue.cc:237] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:08.254854 9904 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:56:08.231765 13061 consensus_meta.cc:287] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.509s user 0.006s sys 0.003s
I20240426 09:56:08.259382 9905 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:08.194237 12871 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:08.257822 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:08.258807 9982 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:08.259878 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:08.220894 12990 raft_consensus.cc:729] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:08.189695 12943 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:08.232425 13072 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:08.219213 12973 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:08.306715 10056 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:08.301045 9935 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:56:08.266544 13098 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
W20240426 09:56:08.330199 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 71501.3us
I20240426 09:56:08.186429 12989 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:08.266031 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:08.303869 9936 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 12.
I20240426 09:56:08.331012 9982 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:08.250377 12900 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:08.249372 12864 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:08.333920 10008 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:56:08.314360 13064 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:08.268486 13061 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:08.354300 12816 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader election won for term 9
I20240426 09:56:08.257488 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:08.258661 12937 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:08.369928 13085 log.cc:924] Time spent T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
I20240426 09:56:08.339717 13098 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:08.374910 12971 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:08.375377 12930 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:08.327821 12965 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:08.376066 12965 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:08.376423 12965 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:08.376793 12965 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:08.379261 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:08.376869 12971 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:08.379693 12816 raft_consensus.cc:686] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:08.369391 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:08.384647 10012 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:08.385532 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:08.366042 13016 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:08.335868 13006 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:08.393343 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 137442us
I20240426 09:56:08.386121 10012 raft_consensus.cc:2454] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:56:08.409948 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:08.322085 13008 consensus_queue.cc:1035] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20240426 09:56:08.412129 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:08.415310 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:08.361078 13093 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:56:08.413422 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 82485.2us
I20240426 09:56:08.423921 10613 raft_consensus.cc:2379] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:08.409528 12816 consensus_queue.cc:237] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:08.373801 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:08.426865 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:08.435673 10084 raft_consensus.cc:2354] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6.
I20240426 09:56:08.436028 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:08.439110 10012 raft_consensus.cc:2379] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 6: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:56:08.437305 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:08.440001 10618 raft_consensus.cc:3037] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Advancing to term 9
I20240426 09:56:08.405793 12964 raft_consensus.cc:2781] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:56:08.376197 13098 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:08.438380 10613 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 11.
I20240426 09:56:08.388520 12864 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:08.376744 13095 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:08.446638 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:08.409466 13023 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:08.417160 13062 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:08.477953 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:08.485879 10012 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 9.
I20240426 09:56:08.478897 9938 raft_consensus.cc:2379] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 12: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:08.436015 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:08.429942 13072 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:08.492273 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:08.488927 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:08.367652 12990 consensus_queue.cc:260] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:08.513561 9936 raft_consensus.cc:2379] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 9: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:08.470970 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:08.511651 10611 raft_consensus.cc:2379] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 7: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:08.485067 12890 raft_consensus.cc:2781] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:56:08.535869 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:08.478602 13098 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:08.410396 13067 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:08.538070 10055 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:08.486397 13055 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 7 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:08.477347 13094 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:08.543318 10616 raft_consensus.cc:2379] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 5: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:56:08.619688 9982 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:08.501338 13023 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:08.703912 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:08.704773 10085 raft_consensus.cc:2379] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 7: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:08.705533 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:08.438992 13093 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:08.717455 13093 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:08.719671 10616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45564 (request call id 599) took 1126 ms. Trace:
I20240426 09:56:08.720099 10616 rpcz_store.cc:276] 0426 09:56:07.593597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:07.597874 (+ 4277us) service_pool.cc:225] Handling call
0426 09:56:08.447721 (+849847us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b5800047a98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce43ee 00000000003399f9 00007f9ca7b13570 00007f9ca7bd9294 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:56:08.719590 (+271869us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3604352}
W20240426 09:56:08.721644 13099 log.cc:924] Time spent T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.207s user 0.000s sys 0.000s
I20240426 09:56:08.540527 12890 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:08.730607 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:08.723250 10016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1875) took 1310 ms. Trace:
I20240426 09:56:08.695950 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:08.659193 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:08.735324 10012 raft_consensus.cc:2379] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 9: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:56:08.689338 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:56:08.723897 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 185904us
I20240426 09:56:08.736124 10055 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:08.726266 12943 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:08.745664 10618 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 9.
I20240426 09:56:08.733314 10016 rpcz_store.cc:276] 0426 09:56:07.413100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:07.413244 (+ 144us) service_pool.cc:225] Handling call
0426 09:56:07.413501 (+ 257us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:07.478511 (+ 65010us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:08.723041 (+1244530us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b54000564d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:08.723200 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42695040}
I20240426 09:56:08.955286 13097 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
W20240426 09:56:09.051195 9905 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.052s sys 0.084s
I20240426 09:56:09.055186 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:56:09.055476 9905 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.614s user 0.052s sys 0.084s
W20240426 09:56:09.055939 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 622026us
I20240426 09:56:08.953341 13115 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 09:56:08.797238 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36762 (request call id 680) took 1022 ms. Trace:
I20240426 09:56:08.794584 13023 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:56:09.016227 12965 consensus_meta.cc:287] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.639s user 0.008s sys 0.001s
W20240426 09:56:08.954748 13068 consensus_meta.cc:287] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.997s user 0.006s sys 0.004s
I20240426 09:56:08.923205 13101 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:56:08.797533 10010 consensus_meta.cc:287] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.606s user 0.004s sys 0.004s
I20240426 09:56:08.888017 12816 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:08.754335 9980 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:56:08.857916 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:08.897205 13111 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
W20240426 09:56:09.065896 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 311623us
W20240426 09:56:08.970546 10056 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.650s user 0.000s sys 0.005s
W20240426 09:56:09.066540 10056 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.746s user 0.000s sys 0.008s
W20240426 09:56:08.922755 13056 log.cc:924] Time spent T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240426 09:56:08.779819 12989 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:09.065081 12816 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:08.894110 12962 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:56:08.991641 12988 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.044s user 0.010s sys 0.006s
I20240426 09:56:08.898274 13086 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:08.820513 12937 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:08.922432 13102 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:56:08.970381 12889 consensus_meta.cc:287] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.985s user 0.001s sys 0.009s
I20240426 09:56:08.824234 13065 raft_consensus.cc:2726] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 09:56:08.951021 13059 consensus_meta.cc:287] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 1.101s user 0.008s sys 0.003s
W20240426 09:56:08.946350 12915 consensus_meta.cc:287] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68: Time spent flushing consensus metadata: real 0.987s user 0.007s sys 0.002s
I20240426 09:56:08.922046 13091 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:08.737479 12933 raft_consensus.cc:2726] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 09:56:09.032735 10089 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 2037) took 1230 ms. Trace:
I20240426 09:56:09.088927 12915 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:09.089279 10089 rpcz_store.cc:276] 0426 09:56:07.802661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:07.944298 (+141637us) service_pool.cc:225] Handling call
0426 09:56:07.999559 (+ 55261us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:07.999959 (+ 400us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:07.999985 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:08.002532 (+ 2547us) log.cc:842] Serialized 26 byte log entry
0426 09:56:08.184947 (+182415us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:56:08.184987 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:08.185053 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:09.032513 (+847460us) raft_consensus.cc:1671] finished
0426 09:56:09.032532 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:09.032673 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8985,"raft.queue_time_us":367,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":26025,"thread_start_us":105196,"threads_started":1,"wal-append.queue_time_us":1016214}
I20240426 09:56:08.722684 13084 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:08.788411 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:08.727684 12888 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:08.624950 12973 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:08.719033 13093 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:08.736119 10087 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 12.
W20240426 09:56:08.796358 13060 consensus_meta.cc:287] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.857s user 0.006s sys 0.003s
I20240426 09:56:09.104823 13060 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:09.105942 10014 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:56:08.515417 13050 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:09.103869 13062 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:08.746832 12890 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:08.659586 12964 raft_consensus.cc:686] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
W20240426 09:56:08.779980 13090 log.cc:924] Time spent T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.466s user 0.001s sys 0.000s
I20240426 09:56:08.830291 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:56:08.781819 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 45779.7us
I20240426 09:56:09.058305 10009 rpcz_store.cc:276] 0426 09:56:07.775210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:07.775333 (+ 123us) service_pool.cc:225] Handling call
0426 09:56:08.797060 (+1021727us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b54000569d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:08.797176 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":922150272}
I20240426 09:56:08.722143 13048 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:08.650210 13094 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.061029 10618 raft_consensus.cc:1261] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:56:09.061411 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:56:09.061672 9906 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.729s user 0.007s sys 0.004s
W20240426 09:56:09.265702 9906 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.933s user 0.008s sys 0.004s
I20240426 09:56:09.265950 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:09.266865 9906 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:09.086421 12889 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.061669 13115 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:09.062506 10586 leader_election.cc:398] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 6. Current term is 7.
I20240426 09:56:09.268841 10586 leader_election.cc:401] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20240426 09:56:09.063412 9979 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.001s sys 0.006s
W20240426 09:56:09.269383 9979 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.828s user 0.001s sys 0.006s
I20240426 09:56:09.064254 10010 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 9.
I20240426 09:56:08.828301 13061 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:09.066743 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 760058us
W20240426 09:56:09.277491 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 215145us
W20240426 09:56:09.071048 10585 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.008s sys 0.011s
I20240426 09:56:08.485694 13095 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.059559 13114 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:09.072805 13111 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:09.080333 13130 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:56:09.073323 12988 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.146s user 0.011s sys 0.008s
I20240426 09:56:09.088280 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:09.078522 13131 raft_consensus.cc:2781] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election won for term 9
I20240426 09:56:09.090602 10617 raft_consensus.cc:2379] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 6: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:56:09.057981 13097 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:09.092715 10089 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 594) took 1061 ms. Trace:
I20240426 09:56:09.093322 12816 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:09.064411 12965 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.066179 13068 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.096446 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 595) took 1064 ms. Trace:
I20240426 09:56:09.096761 12888 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.097424 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:08.591478 13098 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:09.106437 12915 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:09.277073 13115 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 12 FOLLOWER]: Advancing to term 13
I20240426 09:56:09.107847 13062 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:09.113083 10055 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
W20240426 09:56:09.429788 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 316758us
I20240426 09:56:09.430538 10055 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:56:09.431654 10055 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:09.271494 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 2376) took 1094 ms. Trace:
I20240426 09:56:09.276963 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:09.433643 10009 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
W20240426 09:56:09.278961 10585 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.752s user 0.008s sys 0.012s
W20240426 09:56:09.434453 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 907389us
I20240426 09:56:09.435108 10585 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:09.091015 13059 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:09.441871 10007 raft_consensus.cc:2379] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 9: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
W20240426 09:56:09.300403 10583 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 1.707s (SENT) [suppressed 1 similar messages]
I20240426 09:56:09.302485 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:09.442828 10012 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 9.
I20240426 09:56:09.303247 13129 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:09.315297 10089 rpcz_store.cc:276] 0426 09:56:08.031768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:09.091945 (+1060177us) service_pool.cc:225] Handling call
0426 09:56:09.092246 (+ 301us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:09.092407 (+ 161us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:09.092431 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:09.092472 (+ 41us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:56:09.092507 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:09.092556 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:09.092660 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:09.264758 12973 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:09.327273 10011 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:09.264856 13065 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:09.265440 12958 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:09.351668 13111 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:09.478919 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 199607us
I20240426 09:56:09.267608 12914 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:09.283418 12976 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 09:56:09.110517 13093 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:09.276126 12937 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:09.493492 10090 raft_consensus.cc:2354] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 8. Current term is 9.
I20240426 09:56:09.073956 12962 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.386473 10084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 596) took 1293 ms. Trace:
I20240426 09:56:09.352133 13097 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:09.095041 13091 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.079470 13101 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.303998 13061 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:09.310873 10081 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:09.432859 10010 rpcz_store.cc:276] 0426 09:56:08.177739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:08.177864 (+ 125us) service_pool.cc:225] Handling call
0426 09:56:09.271433 (+1093569us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:09.444418 10088 raft_consensus.cc:2379] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 7: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:09.446121 13095 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:09.303135 12894 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:09.466343 13126 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:56:09.352413 9904 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.732s user 0.118s sys 0.159s
I20240426 09:56:09.302686 13122 raft_consensus.cc:2726] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 6. Current term is 7.
I20240426 09:56:09.362782 12968 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:09.495959 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:09.497623 10084 rpcz_store.cc:276] 0426 09:56:08.092738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:09.097646 (+1004908us) service_pool.cc:225] Handling call
0426 09:56:09.097844 (+ 198us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:09.097974 (+ 130us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 09:56:09.108361 (+ 10387us) log.cc:842] Serialized 14 byte log entry
0426 09:56:09.386115 (+277754us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:09.386171 (+ 56us) raft_consensus.cc:1631] Marking committed up to 8
0426 09:56:09.386223 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:09.386289 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:09.386415 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":338246,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":50,"raft.queue_time_us":88,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":86,"thread_start_us":287305,"threads_started":2}
I20240426 09:56:09.083144 13102 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.521039 10009 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 7.
I20240426 09:56:09.447729 13023 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:09.086577 13086 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.310535 13114 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.319033 13130 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.751102 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36762 (request call id 690) took 1226 ms. Trace:
I20240426 09:56:09.751605 10009 rpcz_store.cc:276] 0426 09:56:08.524773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:09.276685 (+751912us) service_pool.cc:225] Handling call
0426 09:56:09.751037 (+474352us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:09.626904 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:09.320394 10085 rpcz_store.cc:276] 0426 09:56:08.032023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:09.092601 (+1060578us) service_pool.cc:225] Handling call
0426 09:56:09.096063 (+ 3462us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:09.096201 (+ 138us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:09.096221 (+ 20us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:09.096264 (+ 43us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:56:09.096292 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:09.096334 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:09.096404 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:56:09.754354 9904 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.134s user 0.122s sys 0.163s
I20240426 09:56:09.485349 12976 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:09.758879 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:09.763042 10089 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 599) took 1408 ms. Trace:
I20240426 09:56:09.763520 10089 rpcz_store.cc:276] 0426 09:56:08.354317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:09.448877 (+1094560us) service_pool.cc:225] Handling call
0426 09:56:09.762559 (+313682us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:09.762720 (+ 161us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:09.762748 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:09.762806 (+ 58us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:56:09.762836 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:09.762886 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:09.762990 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:09.454013 12859 raft_consensus.cc:2781] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:56:09.453711 12964 consensus_queue.cc:237] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:09.769258 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:56:09.799679 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 499270us
I20240426 09:56:09.497416 13111 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:09.766342 10084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 600) took 1409 ms. Trace:
I20240426 09:56:09.475065 12943 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:09.800897 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:09.563302 13044 raft_consensus.cc:2781] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:09.769548 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:09.786384 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:09.485355 12958 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:09.905159 10585 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:09.826175 10089 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 1864) took 1101 ms. Trace:
I20240426 09:56:09.865676 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1893) took 1554 ms. Trace:
I20240426 09:56:09.629166 12988 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:09.908943 10084 rpcz_store.cc:276] 0426 09:56:08.356853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:09.765538 (+1408685us) service_pool.cc:225] Handling call
0426 09:56:09.765814 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:09.765980 (+ 166us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:09.766004 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:09.766042 (+ 38us) raft_consensus.cc:1631] Marking committed up to 8
0426 09:56:09.766076 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:09.766123 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:09.766211 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:09.479960 13065 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.910532 9940 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:56:09.480463 12973 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.746968 13097 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:09.914319 9935 raft_consensus.cc:2379] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 9: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
W20240426 09:56:09.904358 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 472735us
I20240426 09:56:10.026445 10055 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:09.922330 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:09.669765 13107 raft_consensus.cc:2726] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:09.901562 9936 raft_consensus.cc:2354] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 8. Current term is 9.
I20240426 09:56:09.720163 12917 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:09.719791 13109 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:09.669900 13112 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:09.888145 12859 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:09.773618 12976 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:09.921559 12888 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:09.888635 12915 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:09.757601 13124 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:09.562124 12937 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.995879 10084 raft_consensus.cc:1261] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 09:56:09.913590 13111 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:09.669538 13120 consensus_queue.cc:1035] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.753s
I20240426 09:56:09.912379 13095 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
W20240426 09:56:10.027166 13131 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.009s sys 0.022s
I20240426 09:56:09.670413 13108 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:09.478055 13129 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:09.493372 12914 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.764729 12968 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:09.765319 12894 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.921001 12943 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:09.910734 12889 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:09.765038 13023 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:09.772734 13061 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:09.765128 13126 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:10.027482 9938 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:56:09.995977 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:56:10.404511 9905 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.094s user 0.011s sys 0.014s
W20240426 09:56:10.415153 9905 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.105s user 0.018s sys 0.014s
I20240426 09:56:09.921200 10010 rpcz_store.cc:276] 0426 09:56:08.311331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:09.753422 (+1442091us) service_pool.cc:225] Handling call
0426 09:56:09.753677 (+ 255us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:09.753859 (+ 182us) raft_consensus.cc:1520] Early marking committed up to index 9
0426 09:56:09.830530 (+ 76671us) log.cc:842] Serialized 14 byte log entry
0426 09:56:09.865296 (+ 34766us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:09.865361 (+ 65us) raft_consensus.cc:1631] Marking committed up to 9
0426 09:56:09.865406 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:09.865478 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:09.865612 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":133260,"thread_start_us":110798,"threads_started":2,"wal-append.queue_time_us":140923}
I20240426 09:56:10.029803 9941 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:56:10.417747 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:56:10.412324 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 369478us
W20240426 09:56:10.410825 9904 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.006s sys 0.011s
W20240426 09:56:10.423686 9904 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.605s user 0.006s sys 0.012s
W20240426 09:56:10.425832 10056 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.011s sys 0.036s
W20240426 09:56:10.426298 10056 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.512s user 0.011s sys 0.036s
I20240426 09:56:09.995761 13143 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:10.123080 13098 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:10.189170 12969 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:10.447654 12969 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:09.997329 12986 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:10.440032 13143 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:10.454725 12986 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:10.114740 10081 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 7.
I20240426 09:56:10.457067 13126 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:09.926426 13137 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:10.157899 13101 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:10.183549 12887 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 09:56:10.457300 10081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 601) took 1978 ms. Trace:
I20240426 09:56:10.046360 12917 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:10.461066 12887 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:10.116464 13060 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:10.041044 13097 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:10.458442 13137 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:10.466696 10583 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.008s sys 0.004s
I20240426 09:56:10.467578 13023 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:10.273501 12958 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:10.446736 13098 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:10.468429 13154 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:10.475984 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:10.476338 13155 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:09.920423 13044 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:09.921025 10089 rpcz_store.cc:276] 0426 09:56:08.724544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:09.825181 (+1100637us) service_pool.cc:225] Handling call
0426 09:56:09.825676 (+ 495us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:09.825866 (+ 190us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:09.825889 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:09.825930 (+ 41us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:56:09.825966 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:09.826013 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:09.826116 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:09.993575 12988 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:10.828459 12894 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:10.477160 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 597) took 2144 ms. Trace:
W20240426 09:56:10.702741 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 120225us
I20240426 09:56:10.073629 13107 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:10.180303 13122 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 11 FOLLOWER]: Advancing to term 12
W20240426 09:56:10.638954 10055 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.009s sys 0.017s
W20240426 09:56:10.654956 10584 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.167s user 0.007s sys 0.002s
W20240426 09:56:10.817028 10011 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 2388) took 2556 ms (client timeout 1565 ms). Trace:
I20240426 09:56:11.060976 13143 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:10.478297 12943 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:10.287775 12973 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:11.110733 10616 raft_consensus.cc:2454] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 10.
I20240426 09:56:10.460058 10081 rpcz_store.cc:276] 0426 09:56:08.478786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:09.071916 (+593130us) service_pool.cc:225] Handling call
0426 09:56:10.456970 (+1385054us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:10.290084 13102 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:10.289052 13130 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:10.780577 10007 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 2384) took 2556 ms (client timeout 1893 ms). Trace:
W20240426 09:56:10.703016 10054 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.003s sys 0.020s
I20240426 09:56:10.478715 12887 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:10.109344 13139 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:56:10.625339 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 187144us
W20240426 09:56:10.189865 13150 log.cc:924] Time spent T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.088s user 0.000s sys 0.001s
I20240426 09:56:11.129961 9980 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
W20240426 09:56:10.182602 13110 log.cc:924] Time spent T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 1.210s user 0.022s sys 0.037s
W20240426 09:56:10.277027 13132 log.cc:924] Time spent T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20240426 09:56:11.131551 10056 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:10.027797 13123 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:10.479233 13095 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:10.186877 13140 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:10.145280 13093 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:11.113061 10584 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.817s user 0.011s sys 0.024s
W20240426 09:56:10.331522 13084 consensus_meta.cc:287] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 1.237s user 0.009s sys 0.001s
W20240426 09:56:11.043064 13131 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.187s user 0.012s sys 0.028s
I20240426 09:56:10.040789 13068 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:11.108840 10583 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.186s user 0.011s sys 0.044s
W20240426 09:56:11.113248 10055 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.086s user 0.024s sys 0.043s
I20240426 09:56:10.170596 13148 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:10.772330 13061 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:11.105540 13107 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:10.991835 12950 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:56:11.115391 10007 rpcz_store.cc:269] 0426 09:56:08.224270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:08.227128 (+ 2858us) service_pool.cc:225] Handling call
0426 09:56:10.775919 (+2548791us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b540025f654. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:10.780324 (+ 4405us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2740262656}
I20240426 09:56:11.120483 13094 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:10.187552 13065 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:56:11.124397 10054 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.126s user 0.006s sys 0.030s
I20240426 09:56:10.906244 12871 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:10.804915 12890 consensus_meta.cc:287] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 1.459s user 0.007s sys 0.003s
I20240426 09:56:10.125470 13059 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:10.535265 13126 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
W20240426 09:56:10.319229 12991 consensus_meta.cc:287] T 1b76245317c24e129e7f16cdb4d99007 P 1fee3da01ebc463396397057fdb81a68: Time spent flushing consensus metadata: real 1.282s user 0.004s sys 0.004s
I20240426 09:56:10.289345 13114 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:10.822600 12900 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:10.133327 13086 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:10.811923 10617 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45570 (request call id 436) took 2431 ms (client timeout 1595 ms). Trace:
W20240426 09:56:10.780478 13156 log.cc:924] Time spent T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.399s user 0.001s sys 0.000s
I20240426 09:56:10.479401 13064 raft_consensus.cc:2741] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20240426 09:56:10.703809 12859 raft_consensus.cc:3037] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:11.155850 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:10.276166 13138 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:11.058022 13053 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:56:11.107872 10011 rpcz_store.cc:269] 0426 09:56:08.260146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:08.369117 (+108971us) service_pool.cc:225] Handling call
0426 09:56:10.816678 (+2447561us) spinlock_profiling.cc:243] Waited 943 ms on lock 0x7b540020fdd4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:10.816839 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2169926400}
W20240426 09:56:11.129840 13103 log.cc:924] Time spent T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 2.527s user 0.017s sys 0.023s
I20240426 09:56:10.703665 13101 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:11.161639 13044 raft_consensus.cc:3037] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:11.135468 13123 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:10.157258 13091 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:11.163982 10086 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 1853) took 3421 ms. Trace:
I20240426 09:56:11.164486 13098 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:11.169490 13137 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:11.170032 13120 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:10.989881 13048 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:11.176909 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 60500.4us
W20240426 09:56:10.344846 13115 consensus_meta.cc:287] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.868s user 0.009s sys 0.000s
W20240426 09:56:11.184422 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 54626.5us
W20240426 09:56:11.179003 12969 consensus_meta.cc:287] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.715s user 0.006s sys 0.006s
I20240426 09:56:11.218549 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1966) took 1459 ms. Trace:
I20240426 09:56:10.156345 13142 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:10.775559 12915 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:11.175060 12894 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:11.096814 10087 rpcz_store.cc:276] 0426 09:56:08.332613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:09.104206 (+771593us) service_pool.cc:225] Handling call
0426 09:56:09.108617 (+ 4411us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:09.108851 (+ 234us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:09.108867 (+ 16us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:09.109138 (+ 271us) log.cc:842] Serialized 26 byte log entry
0426 09:56:09.362878 (+253740us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:56:09.362926 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:09.363004 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:10.469044 (+1106040us) raft_consensus.cc:1671] finished
0426 09:56:10.469082 (+ 38us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:10.476961 (+ 7879us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":74,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":6116,"thread_start_us":94225,"threads_started":1,"wal-append.queue_time_us":961049}
I20240426 09:56:10.702915 13124 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:10.704201 13112 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:11.184018 13105 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:11.212141 12968 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:11.212630 12958 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:56:11.214134 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 1.18764e+06us
I20240426 09:56:11.201578 12914 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:11.204355 12893 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:11.215480 13148 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:11.216500 9932 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 1955) took 3761 ms (client timeout 3039 ms). Trace:
I20240426 09:56:11.183733 12950 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:10.082068 13109 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:11.209582 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:11.223151 9933 rpcz_store.cc:276] 0426 09:56:09.758762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:09.768263 (+ 9501us) service_pool.cc:225] Handling call
0426 09:56:09.768535 (+ 272us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:10.029856 (+261321us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:10.029884 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:10.029921 (+ 37us) raft_consensus.cc:1631] Marking committed up to 9
0426 09:56:10.029955 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:11.218112 (+1188157us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b54003c4bd4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:11.218173 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:11.218359 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":600567552}
I20240426 09:56:11.172130 13131 raft_consensus.cc:686] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:11.171185 13108 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:11.240293 13100 raft_consensus.cc:2781] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 09:56:11.240748 12893 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:11.189177 10617 rpcz_store.cc:269] 0426 09:56:08.380571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:08.381122 (+ 551us) service_pool.cc:225] Handling call
0426 09:56:10.811486 (+2430364us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b54004029d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:10.811730 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1332658944}
I20240426 09:56:11.241086 13120 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:11.242103 12965 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:11.246878 13053 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:11.236719 9932 rpcz_store.cc:269] 0426 09:56:07.455307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:07.455655 (+ 348us) service_pool.cc:225] Handling call
0426 09:56:11.216108 (+3760453us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400237dd4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:11.216294 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":320130816}
I20240426 09:56:11.208425 13149 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:11.247931 12890 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:11.205142 10086 rpcz_store.cc:276] 0426 09:56:07.742964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:07.743123 (+ 159us) service_pool.cc:225] Handling call
0426 09:56:07.743410 (+ 287us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:07.757670 (+ 14260us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:07.757696 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:07.758059 (+ 363us) log.cc:842] Serialized 26 byte log entry
0426 09:56:09.091892 (+1333833us) spinlock_profiling.cc:243] Waited 7.9 ms on lock 0x7b180032bc00. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fcfdb75 00007f9ca8b37789 00007f9c9fd1b0eb 00007f9c9fd194ef 00007f9c9fd1b652 00007f9ca28c743b 00007f9ca28ccee3 00007f9ca28cd291 00007f9ca2c1a75f
0426 09:56:09.318089 (+226197us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:56:09.318131 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:09.318200 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:11.163477 (+1845277us) raft_consensus.cc:1671] finished
0426 09:56:11.163517 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:11.163808 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13925,"raft.queue_time_us":14237,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":9459,"spinlock_wait_cycles":18175360,"thread_start_us":226019,"threads_started":1,"wal-append.queue_time_us":2370731}
I20240426 09:56:11.235814 13111 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:11.257644 9979 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.760s user 0.011s sys 0.016s
W20240426 09:56:11.260872 9979 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.834s user 0.012s sys 0.016s
I20240426 09:56:11.240365 12917 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:11.256533 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:11.226184 12969 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:11.198554 13154 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:11.249894 13100 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:11.256238 13107 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:56:11.270134 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 138998us
I20240426 09:56:11.256971 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:11.273887 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:11.232429 13101 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:11.268572 13100 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:11.284274 13148 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:11.285168 10016 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:56:11.259042 12887 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:11.275315 13082 log.cc:924] Time spent T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 3.257s user 0.024s sys 0.048s
I20240426 09:56:11.290302 13129 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:11.290915 10011 raft_consensus.cc:1261] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:56:11.251740 13126 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:11.282702 12985 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 3.316s user 0.078s sys 0.098s
I20240426 09:56:11.205935 13155 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:11.270149 13084 raft_consensus.cc:2726] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:11.247781 13140 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:11.299136 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 2396) took 2888 ms. Trace:
I20240426 09:56:11.299890 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:11.299961 10011 rpcz_store.cc:276] 0426 09:56:08.404669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:11.228705 (+2824036us) service_pool.cc:225] Handling call
0426 09:56:11.269783 (+ 41078us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:11.291866 (+ 22083us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:11.292932 (+ 1066us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:11.267323 12894 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:11.279496 13139 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:11.277974 13160 raft_consensus.cc:2726] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 09:56:11.305675 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:11.291042 13138 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:11.298889 12985 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.384s user 0.078s sys 0.099s
W20240426 09:56:11.301788 10585 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.213s user 0.030s sys 0.026s
I20240426 09:56:11.310847 10056 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:11.298403 13095 leader_election.cc:288] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:11.285818 13142 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:11.319082 12914 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:56:11.318681 10585 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.292s user 0.030s sys 0.026s
I20240426 09:56:11.278107 12958 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:11.315794 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 2399) took 2839 ms. Trace:
I20240426 09:56:11.321964 9938 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:11.320798 12914 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:11.320554 13115 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:11.323153 13138 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:11.324133 12914 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:11.324885 13115 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 13 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:11.326339 12915 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:11.326869 13129 consensus_queue.cc:1035] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 09:56:11.321753 10007 rpcz_store.cc:276] 0426 09:56:08.476047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:11.252047 (+2776000us) service_pool.cc:225] Handling call
0426 09:56:11.286802 (+ 34755us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:11.315572 (+ 28770us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:11.315589 (+ 17us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:11.315616 (+ 27us) raft_consensus.cc:1631] Marking committed up to 4
0426 09:56:11.315642 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:11.315679 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:11.315749 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:11.328450 9932 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 5.
I20240426 09:56:11.280191 13105 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:11.326869 10612 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45570 (request call id 429) took 3466 ms (client timeout 2166 ms). Trace:
I20240426 09:56:11.309199 13111 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:11.337832 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:56:11.335505 10612 rpcz_store.cc:269] 0426 09:56:07.860423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:07.873111 (+ 12688us) service_pool.cc:225] Handling call
0426 09:56:11.326470 (+3453359us) spinlock_profiling.cc:243] Waited 359 ms on lock 0x7b540064ee54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:11.326665 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":826361728}
I20240426 09:56:11.338939 9940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1976) took 1458 ms. Trace:
I20240426 09:56:11.344936 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:11.346853 10012 raft_consensus.cc:2454] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:11.338155 13098 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:11.338881 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:11.294499 13149 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:11.827657 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 83062.3us
I20240426 09:56:11.390277 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 687) took 2866 ms. Trace:
W20240426 09:56:11.998041 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 611918us
I20240426 09:56:11.343355 9940 rpcz_store.cc:276] 0426 09:56:09.816364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:10.415484 (+599120us) service_pool.cc:225] Handling call
0426 09:56:10.416079 (+ 595us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:10.416597 (+ 518us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 09:56:10.780478 (+363881us) log.cc:842] Serialized 14 byte log entry
0426 09:56:11.274651 (+494173us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:11.274703 (+ 52us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:56:11.274756 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:11.274857 (+ 101us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:11.274996 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":837431,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":3630,"raft.queue_time_us":7254,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":19468,"spinlock_wait_cycles":13632256,"thread_start_us":726016,"threads_started":2,"wal-append.queue_time_us":515073}
I20240426 09:56:12.000775 10011 rpcz_store.cc:276] 0426 09:56:08.524071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:11.308896 (+2784825us) service_pool.cc:225] Handling call
0426 09:56:11.344642 (+ 35746us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:11.386436 (+ 41794us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:11.386469 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:11.390000 (+ 3531us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:56:11.390047 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:11.390108 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:11.390213 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:11.349723 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:11.362370 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 689) took 2837 ms. Trace:
I20240426 09:56:12.009728 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:11.395156 9939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 2381) took 3648 ms. Trace:
I20240426 09:56:11.547215 10083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 2030) took 3834 ms. Trace:
I20240426 09:56:11.345502 13097 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:11.544299 13166 log.cc:924] Time spent T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.181s user 0.000s sys 0.000s
W20240426 09:56:11.362704 10583 connection.cc:232] RPC call timeout handler was delayed by -1.0253485119999368s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2138s for a response
W20240426 09:56:11.354585 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 1.44142e+06us
I20240426 09:56:11.431579 12985 raft_consensus.cc:3032] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:56:12.008774 10007 rpcz_store.cc:276] 0426 09:56:08.524549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:11.332188 (+2807639us) service_pool.cc:225] Handling call
0426 09:56:11.361790 (+ 29602us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:11.362043 (+ 253us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:11.362067 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:11.362108 (+ 41us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:56:11.362134 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:11.362192 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:11.362303 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:56:11.874064 13100 consensus_meta.cc:287] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.575s user 0.003s sys 0.006s
I20240426 09:56:11.349345 13140 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:11.376327 10614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45570 (request call id 421) took 3704 ms. Trace:
I20240426 09:56:11.397035 13142 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:11.915833 12859 consensus_meta.cc:287] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.667s user 0.008s sys 0.000s
I20240426 09:56:11.347963 13123 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:12.002409 13124 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:11.401036 13107 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:12.011318 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:11.396620 13131 consensus_queue.cc:237] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.035665 10611 raft_consensus.cc:2354] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6.
I20240426 09:56:12.035511 13107 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:12.036180 10611 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45564 (request call id 607) took 1569 ms. Trace:
I20240426 09:56:12.036542 10611 rpcz_store.cc:276] 0426 09:56:10.466949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:10.479617 (+ 12668us) service_pool.cc:225] Handling call
0426 09:56:11.428539 (+948922us) spinlock_profiling.cc:243] Waited 4.17 ms on lock 0x7b5800047a98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce43ee 00000000003399f9 00007f9ca7b13570 00007f9ca7bd9294 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:56:12.036104 (+607565us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9592704}
I20240426 09:56:12.037353 13107 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:12.037858 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.038568 10611 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:56:12.039624 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.040997 10611 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 8.
I20240426 09:56:12.042027 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.036648 12986 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:11.362257 13111 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:12.043280 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:12.044588 13111 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:12.029083 10614 rpcz_store.cc:276] 0426 09:56:07.637708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:07.637885 (+ 177us) service_pool.cc:225] Handling call
0426 09:56:07.638236 (+ 351us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:07.737547 (+ 99311us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:07.737579 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:07.738011 (+ 432us) log.cc:842] Serialized 26 byte log entry
0426 09:56:07.878688 (+140677us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:56:07.920078 (+ 41390us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:11.059206 (+3139128us) spinlock_profiling.cc:243] Waited 40 ms on lock 0x7b54007b2120. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2bd48f1 00007f9ca2bdf3b0 00007f9ca2bdf358 00007f9ca2bdf2d8 00007f9ca2bdf293 00007f9ca2bde51c 00007f9ca8c4ee26 00007f9ca8c47428 00007f9ca2c1b51b 00007f9ca2c1e010
0426 09:56:11.327439 (+268233us) spinlock_profiling.cc:243] Waited 99 ms on lock 0x7b54007b1454. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:11.327498 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:11.330950 (+ 3452us) raft_consensus.cc:1671] finished
0426 09:56:11.330969 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:11.341618 (+ 10649us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":152,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"spinlock_wait_cycles":319611008,"thread_start_us":134683,"threads_started":1,"wal-append.queue_time_us":140705}
I20240426 09:56:12.047333 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:12.003312 13120 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:11.396278 13160 raft_consensus.cc:2781] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:11.480382 12893 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:11.480198 12887 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:56:12.026682 9938 consensus_meta.cc:287] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.695s user 0.009s sys 0.000s
I20240426 09:56:12.027648 13129 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:11.545199 13139 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:11.388733 12969 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:11.362509 12988 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:12.030169 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.011505 13155 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:56:12.028561 13044 consensus_meta.cc:287] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68: Time spent flushing consensus metadata: real 0.814s user 0.006s sys 0.006s
I20240426 09:56:12.048691 13160 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:11.396246 12958 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:12.044590 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.010680 10083 rpcz_store.cc:276] 0426 09:56:07.712292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:07.713363 (+ 1071us) service_pool.cc:225] Handling call
0426 09:56:07.752959 (+ 39596us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:07.757955 (+ 4996us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:07.757981 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:07.758301 (+ 320us) log.cc:842] Serialized 26 byte log entry
0426 09:56:07.876201 (+117900us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:56:08.124494 (+248293us) log.cc:842] Serialized 14 byte log entry
0426 09:56:08.127626 (+ 3132us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:08.595140 (+467514us) spinlock_profiling.cc:243] Waited 4.68 ms on lock 0x7b54001ef5d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:08.595189 (+ 49us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:10.188316 (+1593127us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b48000b0c58. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca7aee2e1 00007f9ca7ae53d7 00007f9ca7ae526a 00007f9ca7bd943f 00007f9ca7bd9107 00007f9ca7bed60c 00007f9ca7bed5b8 00007f9ca7bed538 00007f9ca7bed4f3 00007f9ca7bec79c 00007f9ca6b6f8d6
0426 09:56:10.829418 (+641102us) spinlock_profiling.cc:243] Waited 4.49 ms on lock 0x7b48000b0c58. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca7aee2e1 00007f9ca7ae53d7 00007f9ca7ae526a 00007f9ca7bd943f 00007f9ca7bd9107 00007f9ca7bed60c 00007f9ca7bed5b8 00007f9ca7bed538 00007f9ca7bed4f3 00007f9ca7bec79c 00007f9ca6b6f8d6
0426 09:56:11.105472 (+276054us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:56:11.478216 (+372744us) raft_consensus.cc:1671] finished
0426 09:56:11.478236 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:11.547040 (+ 68804us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":122438,"prepare.queue_time_us":285446,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":13912,"raft.run_cpu_time_us":21114,"raft.run_wall_time_us":5115693,"spinlock_wait_cycles":80280192,"thread_start_us":357196,"threads_started":2,"wal-append.queue_time_us":110375}
I20240426 09:56:12.003000 13108 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:12.022248 12985 raft_consensus.cc:729] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:12.049687 9938 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:56:12.007072 13154 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:12.036756 12976 consensus_meta.cc:287] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 1.142s user 0.009s sys 0.002s
I20240426 09:56:12.027453 13100 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.033725 13105 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:12.049998 13101 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:12.058145 10617 raft_consensus.cc:1261] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:56:12.010845 9939 rpcz_store.cc:276] 0426 09:56:07.747039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:07.747165 (+ 126us) service_pool.cc:225] Handling call
0426 09:56:08.182272 (+435107us) spinlock_profiling.cc:243] Waited 2.82 ms on lock 0x7b5800004598. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:56:08.182445 (+ 173us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:08.182743 (+ 298us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:08.182789 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:08.187808 (+ 5019us) log.cc:842] Serialized 26 byte log entry
0426 09:56:08.339075 (+151267us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:56:08.339115 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:08.339182 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:11.122853 (+2783671us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:56:11.378176 (+255323us) raft_consensus.cc:1671] finished
0426 09:56:11.378195 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:11.394986 (+ 16791us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":27785,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"spinlock_wait_cycles":54256896,"thread_start_us":123862,"threads_started":1,"wal-append.queue_time_us":278931}
I20240426 09:56:12.064528 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.065305 13062 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:12.056425 13129 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:12.059156 12894 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:12.067140 10586 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:12.068890 9940 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:56:12.058665 13160 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:12.034933 12859 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.068800 10615 raft_consensus.cc:1226] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:12.072441 10614 raft_consensus.cc:2379] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 9: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:56:12.077562 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 691) took 3550 ms. Trace:
I20240426 09:56:12.057049 13044 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.081583 13129 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:12.082154 12985 consensus_queue.cc:260] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.075569 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
W20240426 09:56:12.082756 10620 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51158 (request call id 467) took 3933 ms (client timeout 2350 ms). Trace:
W20240426 09:56:12.086740 12950 consensus_meta.cc:287] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68: Time spent flushing consensus metadata: real 0.820s user 0.003s sys 0.007s
I20240426 09:56:12.084122 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.084604 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:12.086674 13069 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:12.095655 10620 rpcz_store.cc:269] 0426 09:56:08.149024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:08.149121 (+ 97us) service_pool.cc:225] Handling call
0426 09:56:12.082426 (+3933305us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b54007733d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:12.082570 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":230206208}
I20240426 09:56:12.066020 12976 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:12.090631 9979 connection.cc:232] RPC call timeout handler was delayed by -1.3732293009999239s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16790000000000002s for a response
W20240426 09:56:12.093405 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 61424.3us
W20240426 09:56:12.088636 10088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 2043) took 3660 ms (client timeout 1679 ms). Trace:
I20240426 09:56:12.099282 10612 raft_consensus.cc:2379] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 6: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:56:12.087136 10585 leader_election.cc:398] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 8. Current term is 9.
I20240426 09:56:12.099998 10585 leader_election.cc:401] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20240426 09:56:12.093276 10011 rpcz_store.cc:276] 0426 09:56:08.524995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:12.020323 (+3495328us) service_pool.cc:225] Handling call
0426 09:56:12.055161 (+ 34838us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:12.071539 (+ 16378us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:12.071554 (+ 15us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:12.074714 (+ 3160us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:56:12.074749 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:12.074841 (+ 92us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:12.074939 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:56:12.083726 10014 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 2385) took 3859 ms (client timeout 2130 ms). Trace:
I20240426 09:56:12.092448 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:12.087647 13149 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:12.099120 9940 raft_consensus.cc:2454] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:56:12.096518 10615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51158 (request call id 465) took 3987 ms. Trace:
W20240426 09:56:12.100827 10014 rpcz_store.cc:269] 0426 09:56:08.224497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:08.236469 (+ 11972us) service_pool.cc:225] Handling call
0426 09:56:12.083410 (+3846941us) spinlock_profiling.cc:243] Waited 769 ms on lock 0x7b540016b054. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:12.083549 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1769320192}
I20240426 09:56:12.109397 13053 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:12.099048 10088 rpcz_store.cc:269] 0426 09:56:08.428302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:08.689058 (+260756us) service_pool.cc:225] Handling call
0426 09:56:12.088222 (+3399164us) spinlock_profiling.cc:243] Waited 708 ms on lock 0x7b54001815d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:12.088410 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1629312896}
I20240426 09:56:12.097962 12950 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.104772 10615 rpcz_store.cc:276] 0426 09:56:08.109261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:08.114195 (+ 4934us) service_pool.cc:225] Handling call
0426 09:56:12.021986 (+3907791us) spinlock_profiling.cc:243] Waited 7.36 ms on lock 0x7b5800047a98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce439c 00007f9c9fce3f5b 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:56:12.059896 (+ 37910us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:12.091092 (+ 31196us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:12.096448 (+ 5356us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16937216}
I20240426 09:56:12.114109 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 2386) took 3878 ms. Trace:
I20240426 09:56:12.114557 12950 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 10 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:12.115499 9941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47242 (request call id 1965) took 2357 ms (client timeout 1679 ms). Trace:
W20240426 09:56:12.115906 9941 rpcz_store.cc:269] 0426 09:56:09.758470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:09.758606 (+ 136us) service_pool.cc:225] Handling call
0426 09:56:12.115183 (+2356577us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b54001973d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:12.115329 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":621598080}
I20240426 09:56:12.116642 13044 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:12.118466 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 2042) took 3690 ms. Trace:
I20240426 09:56:12.118854 10085 rpcz_store.cc:276] 0426 09:56:08.427954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:09.873700 (+1445746us) service_pool.cc:225] Handling call
0426 09:56:09.956722 (+ 83022us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:10.082066 (+125344us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:10.082098 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:10.082141 (+ 43us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:56:10.082177 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:12.118013 (+2035836us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54000d9154. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:12.118075 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:12.118287 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":287734528}
W20240426 09:56:12.120963 10015 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 1889) took 4101 ms (client timeout 3335 ms). Trace:
W20240426 09:56:12.121389 10015 rpcz_store.cc:269] 0426 09:56:08.019658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:08.019774 (+ 116us) service_pool.cc:225] Handling call
0426 09:56:12.120560 (+4100786us) spinlock_profiling.cc:243] Waited 79.7 ms on lock 0x7b54003bd8d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:12.120770 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":183256320}
I20240426 09:56:12.115406 10011 raft_consensus.cc:1261] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:56:12.123678 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1896) took 2220 ms. Trace:
W20240426 09:56:12.127130 10008 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 2381) took 3903 ms (client timeout 3291 ms). Trace:
I20240426 09:56:12.127378 10011 rpcz_store.cc:276] 0426 09:56:09.903077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:12.114596 (+2211519us) service_pool.cc:225] Handling call
0426 09:56:12.114984 (+ 388us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:12.123437 (+ 8453us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:12.123628 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:12.114511 10009 rpcz_store.cc:276] 0426 09:56:08.235983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:09.752320 (+1516337us) service_pool.cc:225] Handling call
0426 09:56:09.752639 (+ 319us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:09.887746 (+135107us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:09.887776 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:09.887816 (+ 40us) raft_consensus.cc:1631] Marking committed up to 4
0426 09:56:09.887853 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:12.113672 (+2225819us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54000587d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:12.113726 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:12.113935 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":310281984}
I20240426 09:56:12.097990 13155 raft_consensus.cc:2781] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 09:56:12.128171 10008 rpcz_store.cc:269] 0426 09:56:08.223481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:08.223611 (+ 130us) service_pool.cc:225] Handling call
0426 09:56:12.126805 (+3903194us) spinlock_profiling.cc:243] Waited 96 ms on lock 0x7b5400174654. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:12.126961 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":220712192}
I20240426 09:56:12.129179 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.130501 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.131314 9940 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:56:12.132236 10014 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1898) took 2228 ms. Trace:
I20240426 09:56:12.132676 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.133378 9940 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 10.
I20240426 09:56:12.128453 13155 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:12.134485 13155 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:12.135818 12967 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:12.136032 12859 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:12.124900 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:12.139858 9941 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:12.143097 13148 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:12.142505 13109 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:12.136860 10014 rpcz_store.cc:276] 0426 09:56:09.903541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:12.125491 (+2221950us) service_pool.cc:225] Handling call
0426 09:56:12.125954 (+ 463us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:12.126390 (+ 436us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:12.126420 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:12.126488 (+ 68us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:56:12.131982 (+ 5494us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:12.132057 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:12.132158 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 09:56:12.099864 9979 connection.cc:232] RPC call timeout handler was delayed by -1.3384274889999688s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1826s for a response
I20240426 09:56:12.138087 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1895) took 2235 ms. Trace:
I20240426 09:56:12.145831 10015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1897) took 2242 ms. Trace:
I20240426 09:56:12.143730 13148 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.149798 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
W20240426 09:56:12.148823 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 85223.2us
I20240426 09:56:12.148561 10007 rpcz_store.cc:276] 0426 09:56:09.902834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:12.051274 (+2148440us) service_pool.cc:225] Handling call
0426 09:56:12.071454 (+ 20180us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:12.104081 (+ 32627us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:12.104108 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:12.112215 (+ 8107us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:56:12.137860 (+ 25645us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:12.137923 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:12.138035 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:12.144631 12985 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:12.105659 9904 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:12.143136 12990 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:12.152969 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.148985 10015 rpcz_store.cc:276] 0426 09:56:09.903316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:12.122165 (+2218849us) service_pool.cc:225] Handling call
0426 09:56:12.122567 (+ 402us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:12.122763 (+ 196us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:12.136481 (+ 13718us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:12.138826 (+ 2345us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:56:12.145606 (+ 6780us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:12.145673 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:12.145777 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:12.158710 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.159617 9936 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 12.
W20240426 09:56:12.163743 9979 connection.cc:232] RPC call timeout handler was delayed by -1.1889429809999683s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21230000000000004s for a response
I20240426 09:56:12.152175 12962 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:12.153910 10055 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:12.154839 13127 raft_consensus.cc:2726] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 8. Current term is 9.
W20240426 09:56:12.164642 10054 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.766s user 0.026s sys 0.076s
I20240426 09:56:12.166981 12976 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:12.169524 10054 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.771s user 0.026s sys 0.076s
W20240426 09:56:12.170838 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 772589us
W20240426 09:56:12.177388 10090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 587) took 4384 ms (client timeout 2191 ms). Trace:
W20240426 09:56:12.193948 10090 rpcz_store.cc:269] 0426 09:56:07.785049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:07.817924 (+ 32875us) service_pool.cc:225] Handling call
0426 09:56:12.152759 (+4334835us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b540037df54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:12.169327 (+ 16568us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3835998976}
I20240426 09:56:12.177537 13128 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader election won for term 9
I20240426 09:56:12.225385 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.228871 9936 raft_consensus.cc:1261] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 09:56:12.178966 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.231948 10612 raft_consensus.cc:2379] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 8: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:56:12.185801 12859 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:12.235939 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 2393) took 3853 ms. Trace:
I20240426 09:56:12.232488 10585 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:12.177909 13100 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:12.236347 10010 rpcz_store.cc:276] 0426 09:56:08.376996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:10.420925 (+2043929us) service_pool.cc:225] Handling call
0426 09:56:10.421384 (+ 459us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:10.951800 (+530416us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:10.951834 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:10.962115 (+ 10281us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:56:10.962154 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:12.230437 (+1268283us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b540025fb54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:12.230480 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:12.230599 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":522354560}
W20240426 09:56:12.233781 10012 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 2387) took 3978 ms (client timeout 3145 ms). Trace:
I20240426 09:56:12.242363 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
W20240426 09:56:12.241117 10012 rpcz_store.cc:269] 0426 09:56:08.255084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:09.443491 (+1188407us) service_pool.cc:225] Handling call
0426 09:56:12.224482 (+2780991us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b540030edd4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:12.233696 (+ 9214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3575757568}
W20240426 09:56:12.249439 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 155756us
I20240426 09:56:12.248135 13148 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:12.248287 13128 raft_consensus.cc:686] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
W20240426 09:56:12.263762 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 115153us
I20240426 09:56:12.261906 13044 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:12.261590 12987 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:12.276371 13119 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:12.271662 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.279412 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:12.267278 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.260977 12859 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:12.285012 10012 raft_consensus.cc:2454] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
W20240426 09:56:12.281495 13122 consensus_meta.cc:287] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68: Time spent flushing consensus metadata: real 1.081s user 0.009s sys 0.008s
I20240426 09:56:12.289717 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.290516 10007 raft_consensus.cc:2379] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 6: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:56:12.278381 13044 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:56:12.297278 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 126312us
I20240426 09:56:12.303207 10010 raft_consensus.cc:2379] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 9: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:56:12.301781 9941 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 10.
I20240426 09:56:12.288286 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.296574 13044 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Advancing to term 9
W20240426 09:56:12.312058 10082 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 584) took 4524 ms (client timeout 1864 ms). Trace:
W20240426 09:56:12.312543 10082 rpcz_store.cc:269] 0426 09:56:07.784315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:07.808111 (+ 23796us) service_pool.cc:225] Handling call
0426 09:56:12.284876 (+4476765us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b5400251a54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:12.308776 (+ 23900us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":609686400}
I20240426 09:56:12.312363 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
W20240426 09:56:12.317692 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 61679.5us
I20240426 09:56:12.284722 13119 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:12.321360 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.298298 13122 raft_consensus.cc:2726] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 09:56:12.325106 13127 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:12.323377 10585 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:56:12.327344 10618 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 12 FOLLOWER]: Advancing to term 13
I20240426 09:56:12.328034 9905 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:56:12.328972 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 271331us
I20240426 09:56:12.325626 13127 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.333537 10013 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:56:12.332656 9980 leader_election.cc:398] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Vote denied by peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313) with higher term. Message: Invalid argument: T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6.
I20240426 09:56:12.335290 9980 leader_election.cc:401] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20240426 09:56:12.335475 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.336843 10010 raft_consensus.cc:2454] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:56:12.338924 13145 raft_consensus.cc:2781] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:12.339358 13145 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:12.339596 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.338080 9905 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:12.337590 9836 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50240 (request call id 316) took 6621 ms. Trace:
I20240426 09:56:12.342352 13062 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:12.339457 13067 raft_consensus.cc:3037] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:12.349897 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.350090 13148 raft_consensus.cc:2781] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:12.346546 13062 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.351552 13148 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:56:12.351686 10613 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51158 (request call id 470) took 1882 ms (client timeout 1687 ms). Trace:
I20240426 09:56:12.340524 13145 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
W20240426 09:56:12.352159 10613 rpcz_store.cc:269] 0426 09:56:10.468941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:10.606927 (+137986us) service_pool.cc:225] Handling call
0426 09:56:12.347957 (+1741030us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b5800047a98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:56:12.351622 (+ 3665us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7077888}
I20240426 09:56:12.341065 9836 rpcz_store.cc:276] 0426 09:56:05.645083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:05.645244 (+ 161us) service_pool.cc:225] Handling call
0426 09:56:12.266070 (+6620826us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:56:05.966617 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:56:06.118729 (+152112us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:56:06.121019 (+ 2290us) write_op.cc:427] Acquired schema lock
0426 09:56:06.126040 (+ 5021us) tablet.cc:660] Decoding operations
0426 09:56:06.209448 (+ 83408us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:56:06.220670 (+ 11222us) write_op.cc:633] Partition lock acquired for write op
0426 09:56:06.228360 (+ 7690us) tablet.cc:683] Acquiring locks for 1 operations
0426 09:56:06.238349 (+ 9989us) tablet.cc:699] Row locks acquired
0426 09:56:06.240373 (+ 2024us) write_op.cc:260] PREPARE: finished
0426 09:56:06.247527 (+ 7154us) write_op.cc:270] Start()
0426 09:56:06.262630 (+ 15103us) write_op.cc:276] Timestamp: P: 1714125366247477 usec, L: 0
0426 09:56:06.287929 (+ 25299us) op_driver.cc:331] REPLICATION: starting
0426 09:56:06.295726 (+ 7797us) log.cc:842] Serialized 536 byte log entry
0426 09:56:06.534912 (+239186us) op_driver.cc:444] REPLICATION: finished
0426 09:56:06.643514 (+108602us) spinlock_profiling.cc:243] Waited 30.1 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9c9fcfde2e 00007f9ca8b37789 00007f9c9fd1b0eb 00007f9c9fd194ef 00007f9c9fd1b652 00007f9ca6657ac0
0426 09:56:07.178754 (+535240us) write_op.cc:301] APPLY: starting
0426 09:56:07.181770 (+ 3016us) tablet.cc:1365] starting BulkCheckPresence
0426 09:56:07.202563 (+ 20793us) tablet.cc:1368] finished BulkCheckPresence
0426 09:56:07.212555 (+ 9992us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:56:07.318748 (+106193us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:56:07.339075 (+ 20327us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0426 09:56:07.345881 (+ 6806us) write_op.cc:312] APPLY: finished
0426 09:56:07.362692 (+ 16811us) log.cc:842] Serialized 23 byte log entry
0426 09:56:07.398617 (+ 35925us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:56:07.413606 (+ 14989us) write_op.cc:446] Released schema lock
0426 09:56:07.414987 (+ 1381us) write_op.cc:341] FINISH: Updating metrics
0426 09:56:07.429178 (+ 14191us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:56:07.431970 (+ 2792us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":251857,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":534648,"apply.run_cpu_time_us":2527,"apply.run_wall_time_us":343687,"num_ops":1,"prepare.queue_time_us":108,"prepare.run_cpu_time_us":3003,"prepare.run_wall_time_us":346852,"replication_time_us":239184,"spinlock_wait_cycles":69237632,"thread_start_us":375904,"threads_started":1}]]}
I20240426 09:56:12.351966 10012 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
W20240426 09:56:12.356413 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 76606.7us
I20240426 09:56:12.352030 13122 raft_consensus.cc:2781] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:56:12.363794 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:56:12.365890 10016 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 2392) took 4057 ms (client timeout 2978 ms). Trace:
W20240426 09:56:12.366356 10016 rpcz_store.cc:269] 0426 09:56:08.376590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:09.265695 (+889105us) service_pool.cc:225] Handling call
0426 09:56:12.365649 (+3099954us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x7b54003bddd4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:12.365815 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4576482048}
I20240426 09:56:12.369086 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 593) took 3277 ms. Trace:
I20240426 09:56:12.369472 9934 rpcz_store.cc:276] 0426 09:56:09.093314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:09.093436 (+ 122us) service_pool.cc:225] Handling call
0426 09:56:09.093674 (+ 238us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:09.351195 (+257521us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:09.351233 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:09.351272 (+ 39us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:56:09.351302 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:12.368716 (+3017414us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b540019c154. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:12.368775 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:12.368936 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":591802624}
I20240426 09:56:12.370888 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:12.372059 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.372452 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:12.373674 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
W20240426 09:56:12.353157 13087 log.cc:924] Time spent T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 4.173s user 0.021s sys 0.032s
I20240426 09:56:12.348516 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:12.351975 13148 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:12.395537 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.378897 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:12.379433 10010 raft_consensus.cc:2454] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:56:12.372834 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 2410) took 1136 ms. Trace:
I20240426 09:56:12.384217 13127 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:12.384840 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.385102 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.386927 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:12.388710 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40928 (request call id 584) took 4599 ms. Trace:
I20240426 09:56:12.389678 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.320302 13128 consensus_queue.cc:237] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:12.391073 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:12.392691 13155 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.385179 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.400934 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:12.400061 10011 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 7.
I20240426 09:56:12.399591 10618 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 13.
I20240426 09:56:12.404723 10010 raft_consensus.cc:1226] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20240426 09:56:12.402479 10008 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 7.
I20240426 09:56:12.411149 12859 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:12.411854 13072 consensus_queue.cc:1035] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 09:56:12.412900 12963 consensus_queue.cc:1035] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 09:56:12.396237 13122 raft_consensus.cc:686] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 7 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:56:12.415839 13122 consensus_queue.cc:237] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.417630 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.419486 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.419802 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.420598 10082 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:12.421469 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:12.422113 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:12.422399 10009 raft_consensus.cc:2379] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 10: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:56:12.423318 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.423602 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.424810 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.425393 10009 raft_consensus.cc:2454] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:56:12.419572 12859 consensus_queue.cc:1035] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20240426 09:56:12.426999 10008 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:56:12.421386 10613 raft_consensus.cc:1226] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 9. Current term is 10. Ops: []
I20240426 09:56:12.428876 13126 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:12.431597 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.432344 10009 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:56:12.435428 9939 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:12.437074 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:12.437628 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:12.438208 9932 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:12.400486 9937 rpcz_store.cc:276] 0426 09:56:07.789610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:07.789897 (+ 287us) service_pool.cc:225] Handling call
0426 09:56:07.790184 (+ 287us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:07.798558 (+ 8374us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:07.798586 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:07.799075 (+ 489us) log.cc:842] Serialized 26 byte log entry
0426 09:56:08.029276 (+230201us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:56:08.029313 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:08.029383 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:12.277547 (+4248164us) spinlock_profiling.cc:243] Waited 6.11 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:56:12.388356 (+110809us) raft_consensus.cc:1671] finished
0426 09:56:12.388375 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:12.388557 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8033,"raft.queue_time_us":17602,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":2863,"spinlock_wait_cycles":14058752,"thread_start_us":229834,"threads_started":1,"wal-append.queue_time_us":322054}
I20240426 09:56:12.437650 10015 raft_consensus.cc:3037] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:12.397363 9933 rpcz_store.cc:276] 0426 09:56:11.239995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:12.042030 (+802035us) service_pool.cc:225] Handling call
0426 09:56:12.042335 (+ 305us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:12.042763 (+ 428us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:12.050238 (+ 7475us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:12.077354 (+ 27116us) log.cc:842] Serialized 26 byte log entry
0426 09:56:12.317185 (+239831us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:56:12.317211 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:12.317253 (+ 42us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:12.372558 (+ 55305us) raft_consensus.cc:1671] finished
0426 09:56:12.372582 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:12.372767 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":119,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":119,"thread_start_us":229706,"threads_started":1,"wal-append.queue_time_us":254995}
I20240426 09:56:12.444701 12937 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:12.390368 13160 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:12.447935 9838 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58806 (request call id 171) took 6686 ms. Trace:
I20240426 09:56:09.102348 8758 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 09:56:12.449406 13064 consensus_queue.cc:1046] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 09:56:12.450273 10007 raft_consensus.cc:3037] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 6
I20240426 09:56:12.450891 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.451390 13016 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:12.452304 9835 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50254 (request call id 326) took 5974 ms. Trace:
I20240426 09:56:12.454038 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:12.449177 9934 raft_consensus.cc:2379] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 6: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:12.459897 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.461211 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:56:12.465162 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 62958.8us
I20240426 09:56:12.467058 10009 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:12.465284 10007 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 6.
I20240426 09:56:12.471045 10611 raft_consensus.cc:2379] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 6: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:56:12.473830 13062 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:12.475350 13049 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:12.448314 9838 rpcz_store.cc:276] 0426 09:56:05.761312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:05.761554 (+ 242us) service_pool.cc:225] Handling call
0426 09:56:12.447777 (+6686223us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:56:06.296145 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:56:06.296626 (+ 481us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:56:06.296651 (+ 25us) write_op.cc:427] Acquired schema lock
0426 09:56:06.296667 (+ 16us) tablet.cc:660] Decoding operations
0426 09:56:06.296969 (+ 302us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:56:06.297012 (+ 43us) write_op.cc:633] Partition lock acquired for write op
0426 09:56:06.297042 (+ 30us) tablet.cc:683] Acquiring locks for 1 operations
0426 09:56:06.297197 (+ 155us) tablet.cc:699] Row locks acquired
0426 09:56:06.297212 (+ 15us) write_op.cc:260] PREPARE: finished
0426 09:56:06.297342 (+ 130us) write_op.cc:270] Start()
0426 09:56:06.297442 (+ 100us) write_op.cc:276] Timestamp: P: 1714125366297291 usec, L: 0
0426 09:56:06.297459 (+ 17us) op_driver.cc:331] REPLICATION: starting
0426 09:56:06.297954 (+ 495us) log.cc:842] Serialized 536 byte log entry
0426 09:56:07.010337 (+712383us) op_driver.cc:444] REPLICATION: finished
0426 09:56:07.441362 (+431025us) write_op.cc:301] APPLY: starting
0426 09:56:07.441438 (+ 76us) tablet.cc:1365] starting BulkCheckPresence
0426 09:56:07.441525 (+ 87us) tablet.cc:1368] finished BulkCheckPresence
0426 09:56:07.441633 (+ 108us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:56:07.441848 (+ 215us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:56:07.441997 (+ 149us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0426 09:56:07.442041 (+ 44us) write_op.cc:312] APPLY: finished
0426 09:56:07.442216 (+ 175us) log.cc:842] Serialized 23 byte log entry
0426 09:56:07.442377 (+ 161us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:56:07.442446 (+ 69us) write_op.cc:446] Released schema lock
0426 09:56:07.442484 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0426 09:56:07.442741 (+ 257us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:56:07.445763 (+ 3022us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":4937906,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":361852,"apply.run_cpu_time_us":1669,"apply.run_wall_time_us":11230,"mutex_wait_us":13337,"num_ops":1,"prepare.queue_time_us":308882,"prepare.run_cpu_time_us":2001,"prepare.run_wall_time_us":2001,"replication_time_us":701778,"thread_start_us":432531,"threads_started":1}]]}
I20240426 09:56:12.475904 13049 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:12.468261 13058 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:12.477936 9940 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:56:12.452661 9835 rpcz_store.cc:276] 0426 09:56:06.477729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:06.478671 (+ 942us) service_pool.cc:225] Handling call
0426 09:56:12.452120 (+5973449us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:56:06.715248 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:56:06.957007 (+241759us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:56:06.971226 (+ 14219us) write_op.cc:427] Acquired schema lock
0426 09:56:06.971258 (+ 32us) tablet.cc:660] Decoding operations
0426 09:56:07.183780 (+212522us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:56:07.189900 (+ 6120us) write_op.cc:633] Partition lock acquired for write op
0426 09:56:07.191114 (+ 1214us) tablet.cc:683] Acquiring locks for 2 operations
0426 09:56:07.330929 (+139815us) tablet.cc:699] Row locks acquired
0426 09:56:07.334240 (+ 3311us) write_op.cc:260] PREPARE: finished
0426 09:56:07.352729 (+ 18489us) write_op.cc:270] Start()
0426 09:56:07.362836 (+ 10107us) write_op.cc:276] Timestamp: P: 1714125367348282 usec, L: 0
0426 09:56:07.362862 (+ 26us) op_driver.cc:331] REPLICATION: starting
0426 09:56:07.613153 (+250291us) log.cc:842] Serialized 919 byte log entry
0426 09:56:07.615679 (+ 2526us) op_driver.cc:444] REPLICATION: finished
0426 09:56:07.616296 (+ 617us) write_op.cc:301] APPLY: starting
0426 09:56:07.616370 (+ 74us) tablet.cc:1365] starting BulkCheckPresence
0426 09:56:07.616477 (+ 107us) tablet.cc:1368] finished BulkCheckPresence
0426 09:56:07.616583 (+ 106us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:56:07.616882 (+ 299us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:56:07.617037 (+ 155us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0426 09:56:07.617086 (+ 49us) write_op.cc:312] APPLY: finished
0426 09:56:07.617251 (+ 165us) log.cc:842] Serialized 29 byte log entry
0426 09:56:07.617416 (+ 165us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:56:07.635253 (+ 17837us) write_op.cc:446] Released schema lock
0426 09:56:07.635298 (+ 45us) write_op.cc:341] FINISH: Updating metrics
0426 09:56:07.635558 (+ 260us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:56:07.635586 (+ 28us) write_op.cc:446] Released schema lock
0426 09:56:08.687769 (+1052183us) spinlock_profiling.cc:243] Waited 248 ms on lock 0x7b54000041d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c42328 00007f9ca665718e 00007f9ca6656968 00007f9ca665b251 00007f9ca665b209 00007f9ca665b199 00007f9ca665b161 00007f9ca665a45d 00007f9ca8f8fd34 00007f9ca8f8fb69
Metrics: {"mutex_wait_us":4802575,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":196,"apply.run_cpu_time_us":1833,"apply.run_wall_time_us":19516,"num_ops":2,"prepare.queue_time_us":94,"prepare.run_cpu_time_us":22439,"prepare.run_wall_time_us":1977471,"replication_time_us":250793,"spinlock_wait_cycles":570054528}]]}
I20240426 09:56:12.477664 13058 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.466729 13112 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:12.473852 9839 catalog_manager.cc:5484] T f46890d4f3474e88835f7b3e08a30e1a P 826f03b2fb9c4b9483623b041a1bf80e reported cstate change: term changed from 6 to 7, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 7 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: UNKNOWN } } }
I20240426 09:56:12.474390 13123 consensus_queue.cc:1035] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:56:12.480577 13058 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:12.458520 12976 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:12.494391 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.501482 10007 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:12.501384 10015 raft_consensus.cc:2454] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:12.503727 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.507083 10054 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:12.507416 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.509533 10619 raft_consensus.cc:1261] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 09:56:12.510159 10619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51158 (request call id 471) took 1901 ms. Trace:
I20240426 09:56:12.510468 10619 rpcz_store.cc:276] 0426 09:56:10.649248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:10.703174 (+ 53926us) service_pool.cc:225] Handling call
0426 09:56:11.429368 (+726194us) spinlock_profiling.cc:243] Waited 4.82 ms on lock 0x7b5800047a98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:56:12.508926 (+1079558us) spinlock_profiling.cc:243] Waited 1.02 ms on lock 0x7b5800047a98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce439c 00007f9c9fce3f5b 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:56:12.509098 (+ 172us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:12.509922 (+ 824us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:12.510089 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13423232}
I20240426 09:56:12.511737 12991 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:45570 (request call id 406) took 6946 ms. Trace:
I20240426 09:56:12.511984 13160 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:12.512249 12937 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:12.512676 13160 raft_consensus.cc:2726] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:56:12.513095 12890 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:12.509765 13049 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:12.514730 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.515702 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.516404 10014 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 9.
I20240426 09:56:12.518857 9932 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:12.538154 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 2413) took 1205 ms. Trace:
I20240426 09:56:12.538599 9932 rpcz_store.cc:276] 0426 09:56:11.314556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:11.386583 (+ 72027us) service_pool.cc:225] Handling call
0426 09:56:12.437495 (+1050912us) spinlock_profiling.cc:243] Waited 6.69 ms on lock 0x7b5800004598. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:56:12.519393 (+ 81898us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15382784}
I20240426 09:56:12.543807 10585 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:56:12.545248 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.550022 13067 raft_consensus.cc:2726] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313) with higher term. Message: Invalid argument: T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6.
I20240426 09:56:12.550715 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.552244 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:56:12.553036 10053 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.8.141.190:45863, user_credentials={real_user=slave}} blocked reactor thread for 48575.4us
I20240426 09:56:12.554790 13152 consensus_queue.cc:1035] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 09:56:12.555270 9934 raft_consensus.cc:1226] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 8. Current term is 9. Ops: []
I20240426 09:56:12.557828 10009 raft_consensus.cc:2454] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:56:12.559964 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.501211 12987 raft_consensus.cc:2726] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:12.556306 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.561647 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.559502 10015 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
W20240426 09:56:12.563412 9979 connection.cc:232] RPC call timeout handler was delayed by -1.0762616360000266s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16210000000000002s for a response
I20240426 09:56:12.564374 10585 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:12.546365 10008 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:56:12.562228 10007 raft_consensus.cc:2454] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:56:12.571676 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.573312 13014 raft_consensus.cc:2781] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:12.573769 13014 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:12.512753 12937 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.574388 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.574000 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:12.570712 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.578331 10585 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:12.579615 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:12.580238 10007 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:12.580796 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:12.581513 10085 raft_consensus.cc:2379] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 7: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:12.581818 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.582428 10015 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 9.
I20240426 09:56:12.583251 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:12.584254 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.584060 13145 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.584950 10015 raft_consensus.cc:2379] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 13: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:56:12.585964 13145 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:12.586668 13155 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:12.587334 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:12.536386 12976 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.579615 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.592761 9906 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:12.518514 12991 rpcz_store.cc:276] 0426 09:56:05.565087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:05.565542 (+ 455us) service_pool.cc:225] Handling call
0426 09:56:06.166077 (+600535us) ts_tablet_manager.cc:903] T 1b76245317c24e129e7f16cdb4d99007 P 1fee3da01ebc463396397057fdb81a68: Initiating tablet copy from peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
0426 09:56:08.121723 (+1955646us) tablet_copy_client.cc:353] Tablet copy session begun
0426 09:56:09.027717 (+905994us) tablet_metadata.cc:641] Metadata flushed
0426 09:56:09.028312 (+ 595us) tablet_copy_client.cc:482] Wrote new tablet metadata
0426 09:56:11.246823 (+2218511us) tablet_copy_client.cc:403] Wrote new consensus metadata
0426 09:56:11.247517 (+ 694us) tablet_replica.cc:205] Creating consensus instance
0426 09:56:12.511420 (+1263903us) ts_tablet_manager.cc:958] Replica registered.
0426 09:56:12.511528 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":86810,"tablet-copy.queue_time_us":414698,"thread_start_us":1104926,"threads_started":5}
I20240426 09:56:12.594075 10009 raft_consensus.cc:1261] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 09:56:12.577107 10088 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 9.
I20240426 09:56:12.595157 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.596150 10083 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:56:12.604509 10014 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:12.597220 13044 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.597934 13119 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:12.598201 10012 raft_consensus.cc:2454] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 6.
I20240426 09:56:12.567456 12950 consensus_queue.cc:1046] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.031s
I20240426 09:56:12.598405 9938 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 8.
I20240426 09:56:12.598096 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:12.580401 10086 raft_consensus.cc:2354] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 9. Current term is 10.
W20240426 09:56:12.573777 13174 log.cc:924] Time spent T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.179s user 0.001s sys 0.000s
W20240426 09:56:12.617693 10618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45564 (request call id 614) took 1281 ms (client timeout 1699 ms). Trace:
I20240426 09:56:12.618434 10008 raft_consensus.cc:2379] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 6: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
W20240426 09:56:12.618144 10618 rpcz_store.cc:269] 0426 09:56:11.336653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:11.339317 (+ 2664us) service_pool.cc:225] Handling call
0426 09:56:12.617539 (+1278222us) spinlock_profiling.cc:243] Waited 323 ms on lock 0x7b5400774cd4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:12.617639 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":742329728}
I20240426 09:56:12.579103 13014 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:12.623646 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.598090 13122 raft_consensus.cc:2781] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:12.668857 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:56:12.669137 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 76476.9us
I20240426 09:56:12.645198 10007 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 10.
I20240426 09:56:12.675798 12991 tablet_copy_client.cc:783] T 1b76245317c24e129e7f16cdb4d99007 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 0 data blocks...
I20240426 09:56:12.676169 10090 raft_consensus.cc:1261] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:56:12.676548 12991 tablet_copy_client.cc:647] T 1b76245317c24e129e7f16cdb4d99007 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Starting download of 1 WAL segments...
I20240426 09:56:12.677152 13152 raft_consensus.cc:2726] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 09:56:12.669828 10620 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 9.
I20240426 09:56:12.679405 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.681383 10007 raft_consensus.cc:2379] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 13: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:56:12.685600 12976 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:12.687238 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:12.681995 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.690443 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.707161 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.708392 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.709026 10089 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:56:12.709388 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.710090 10014 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 9
I20240426 09:56:12.710562 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.711246 10085 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:56:12.672317 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:12.677084 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.713346 9906 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:12.691709 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.691893 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47242 (request call id 1971) took 2788 ms. Trace:
I20240426 09:56:12.671020 13122 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
W20240426 09:56:12.711145 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 138694us
I20240426 09:56:12.712194 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.716996 10612 raft_consensus.cc:2379] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 6: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:56:12.717136 10585 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:12.693199 10056 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:12.694273 13016 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:12.719105 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.718988 13016 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:12.700583 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.720233 10010 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:56:12.699338 12937 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:12.721231 10010 raft_consensus.cc:1226] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 10. Current term is 11. Ops: []
I20240426 09:56:12.721665 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.705439 13044 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:12.708914 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:12.723313 10083 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:12.723654 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.724202 10007 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:12.719311 10009 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 10.
I20240426 09:56:12.721072 13016 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:12.721266 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.717191 9980 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:56:12.706794 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.700747 10620 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20240426 09:56:12.729645 9906 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:12.727174 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:12.732992 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:12.733608 10008 raft_consensus.cc:3037] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:12.734189 9932 raft_consensus.cc:2354] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 9. Current term is 10.
I20240426 09:56:12.597824 13148 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.691566 12940 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:56:12.735559 10087 raft_consensus.cc:2379] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 6: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:12.727146 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.753860 9906 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:12.727360 13122 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:12.756595 10586 leader_election.cc:398] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 election: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 9. Current term is 10.
I20240426 09:56:12.759667 10586 leader_election.cc:401] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20240426 09:56:12.729290 12976 raft_consensus.cc:2781] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:12.740258 12937 raft_consensus.cc:2741] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20240426 09:56:12.767266 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.754204 13044 raft_consensus.cc:2781] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:56:12.714471 9935 rpcz_store.cc:276] 0426 09:56:09.815162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:10.411057 (+595895us) service_pool.cc:225] Handling call
0426 09:56:10.412116 (+ 1059us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:10.797103 (+384987us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 09:56:11.190704 (+393601us) log.cc:842] Serialized 14 byte log entry
0426 09:56:11.283107 (+ 92403us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:11.283155 (+ 48us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:56:11.283195 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:12.603189 (+1319994us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b5400100ed4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:12.603233 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:12.603357 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2395,"prepare.queue_time_us":467086,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":3234,"raft.queue_time_us":1817,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":174,"spinlock_wait_cycles":837691648,"thread_start_us":449955,"threads_started":2,"wal-append.queue_time_us":111903,"wal-append.run_cpu_time_us":1338,"wal-append.run_wall_time_us":1020446}
I20240426 09:56:12.768306 12859 raft_consensus.cc:2781] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:56:12.760918 13152 raft_consensus.cc:2781] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:12.782471 10083 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 9.
I20240426 09:56:12.766954 12947 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:12.789029 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.799652 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:12.786692 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.787237 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:12.767524 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.783242 10056 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:12.751124 12917 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20240426 09:56:12.769786 12976 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:12.773777 13117 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:12.801016 10016 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 9.
I20240426 09:56:12.803686 10010 raft_consensus.cc:2354] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 9. Current term is 10.
I20240426 09:56:12.816207 9906 leader_election.cc:302] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:12.785642 12969 raft_consensus.cc:2781] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:12.785880 13044 raft_consensus.cc:686] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [term 6 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:56:12.823458 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.825851 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:12.793906 13152 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:12.827927 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.829097 13016 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
W20240426 09:56:12.829375 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 57387.3us
I20240426 09:56:12.829566 12887 raft_consensus.cc:2781] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:12.830085 10056 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:12.830047 12887 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:12.829802 13016 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:12.836552 12887 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:12.833459 9935 raft_consensus.cc:2379] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 9: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:12.833575 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.840682 9932 raft_consensus.cc:2379] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 10: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:12.835000 9906 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:12.828560 9937 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:56:12.804319 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:12.794142 12859 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:12.820426 12976 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:12.846033 9906 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:12.836505 12990 raft_consensus.cc:2726] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:56:12.823009 12969 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:12.848979 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.831758 13152 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:12.867185 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.867889 9937 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:56:12.816383 13129 consensus_queue.cc:1046] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20240426 09:56:12.871170 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.865861 9940 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:56:12.846850 13046 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:12.847297 10617 raft_consensus.cc:2379] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 10: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:56:12.822209 13117 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:12.898288 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.894960 9906 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:12.862176 12859 raft_consensus.cc:3037] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:12.861045 12963 raft_consensus.cc:2781] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:56:12.872606 12969 raft_consensus.cc:3037] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:12.893822 12987 raft_consensus.cc:2781] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:12.850178 13064 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:12.896688 13146 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 09:56:12.908897 9906 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:12.911214 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:12.912786 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.918555 9940 raft_consensus.cc:2396] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
W20240426 09:56:12.929466 9982 leader_election.cc:334] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113): Timed out: RequestConsensusVote RPC to 127.8.141.129:38113 timed out after 1.679s (SENT)
I20240426 09:56:12.936039 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.904858 10616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51158 (request call id 475) took 1513 ms. Trace:
I20240426 09:56:12.940483 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.941073 9906 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:12.911849 13145 raft_consensus.cc:2741] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20240426 09:56:12.907366 12963 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:12.907855 13148 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:12.918963 13183 log.cc:924] Time spent T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20240426 09:56:12.914415 13146 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:12.923632 12990 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 09:56:12.950663 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.952622 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:12.951865 13146 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:12.940207 10616 rpcz_store.cc:276] 0426 09:56:11.388926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:11.429119 (+ 40193us) service_pool.cc:225] Handling call
0426 09:56:12.801355 (+1372236us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b5800047a98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:56:12.902701 (+101346us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8378624}
I20240426 09:56:12.924293 13016 leader_election.cc:288] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:12.971120 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:56:12.974417 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 51518.1us
I20240426 09:56:12.975121 9906 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:12.984022 10087 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:12.987244 9906 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:12.949232 12963 raft_consensus.cc:3037] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
W20240426 09:56:12.990149 10583 leader_election.cc:334] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 1.707s (SENT)
I20240426 09:56:12.912142 13046 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:12.953174 12990 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:12.972237 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:12.996090 9906 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:12.996937 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:12.946398 13044 consensus_queue.cc:237] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:12.996754 10013 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34378 (request call id 2373) took 5314 ms (client timeout 1913 ms). Trace:
I20240426 09:56:13.007493 9906 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:13.008441 9906 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:56:12.993480 13149 raft_consensus.cc:2781] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:56:13.012177 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:13.012507 12903 raft_consensus.cc:3032] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 09:56:13.012948 12903 raft_consensus.cc:729] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
W20240426 09:56:13.013151 12983 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.012s sys 0.007s
W20240426 09:56:13.013484 12983 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.564s user 0.012s sys 0.007s
I20240426 09:56:13.014199 13148 raft_consensus.cc:2781] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:13.014573 13148 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:13.013689 12903 consensus_queue.cc:260] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:13.015089 12985 raft_consensus.cc:2781] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:56:13.015496 12985 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:13.015839 12985 raft_consensus.cc:3037] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:56:12.997764 12990 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:13.026149 12947 raft_consensus.cc:2726] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 9. Current term is 10.
W20240426 09:56:13.027216 13186 log.cc:924] Time spent T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.132s user 0.001s sys 0.000s
I20240426 09:56:12.991073 13123 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20240426 09:56:13.007545 10013 rpcz_store.cc:269] 0426 09:56:07.680911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:07.681127 (+ 216us) service_pool.cc:225] Handling call
0426 09:56:12.987801 (+5306674us) spinlock_profiling.cc:243] Waited 4.63 s on lock 0x7b540030fcd4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:12.994046 (+ 6245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10644803968}
I20240426 09:56:12.942428 13064 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:13.037449 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 120967us
I20240426 09:56:13.014935 13148 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:13.042100 10014 raft_consensus.cc:1261] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 09:56:13.044610 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:56:13.045413 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 55287.4us
I20240426 09:56:12.999343 13129 raft_consensus.cc:2726] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:56:13.052446 13149 raft_consensus.cc:686] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:13.053644 10585 leader_election.cc:398] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 election: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 9. Current term is 10.
I20240426 09:56:13.053465 13149 consensus_queue.cc:237] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.035753 13123 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:13.056718 13046 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:13.059041 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:13.061002 10008 raft_consensus.cc:2454] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 10.
I20240426 09:56:13.061602 12887 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.061965 13117 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:13.063238 10585 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:56:13.080132 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.081460 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.082245 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.082659 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.082995 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.083410 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.083865 9936 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:56:13.082155 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.085445 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.087538 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.088165 9934 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 9.
I20240426 09:56:13.063385 13145 raft_consensus.cc:2781] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:56:13.088884 13145 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:13.089315 13145 raft_consensus.cc:3037] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:56:13.081724 12990 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.099437 10056 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:13.103986 13152 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.095762 13046 consensus_queue.cc:1035] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:56:13.065032 13123 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:13.101971 13122 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:13.097344 12859 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.088986 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:13.116214 9932 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 12 FOLLOWER]: Advancing to term 13
I20240426 09:56:13.097186 13049 raft_consensus.cc:3032] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 09:56:13.094259 12987 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 09:56:13.100724 13188 log.cc:924] Time spent T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
I20240426 09:56:13.089423 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:56:13.109490 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 113782us
I20240426 09:56:13.111771 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.119336 9934 raft_consensus.cc:2379] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 10: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:13.121165 10619 raft_consensus.cc:1226] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 5. Current term is 6. Ops: []
I20240426 09:56:13.105222 13196 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
W20240426 09:56:13.121954 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 67594.3us
I20240426 09:56:13.125113 9937 raft_consensus.cc:2354] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8.
I20240426 09:56:13.112334 13064 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:13.128034 10620 raft_consensus.cc:1226] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 9. Current term is 10. Ops: []
I20240426 09:56:13.128588 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.134635 13194 raft_consensus.cc:2781] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader election won for term 10
I20240426 09:56:13.117586 13049 raft_consensus.cc:729] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:13.135993 13049 consensus_queue.cc:260] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.136520 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.138346 12859 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:13.141209 13122 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:13.151857 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.153404 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:13.154443 12990 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:13.155371 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 90529.4us
I20240426 09:56:13.071219 13014 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:13.120052 12987 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:13.157604 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:56:13.158397 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 150013us
I20240426 09:56:13.159312 9906 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:56:13.161849 10055 connection.cc:232] RPC call timeout handler was delayed by -3.1227906469998743s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19130000000000003s for a response
I20240426 09:56:13.162345 9906 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:13.162861 10585 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:13.151998 10616 raft_consensus.cc:1226] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:13.136957 13049 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:13.169624 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.178400 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.180619 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.181703 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.169910 13117 raft_consensus.cc:2781] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:13.183320 13117 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:13.183728 13117 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:13.170461 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:13.167171 13193 raft_consensus.cc:2726] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20240426 09:56:13.170944 10583 leader_election.cc:334] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 1.864s (SENT)
I20240426 09:56:13.171343 13014 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:13.172408 12887 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:13.176190 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:13.186497 9935 raft_consensus.cc:2354] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6.
I20240426 09:56:13.186504 10089 raft_consensus.cc:2354] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6.
I20240426 09:56:13.176431 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:13.187644 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.188187 9937 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:13.177382 13152 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:13.158452 13136 raft_consensus.cc:2781] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:56:13.170729 13046 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:13.189352 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:13.190995 10008 raft_consensus.cc:1261] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:56:13.195343 10085 raft_consensus.cc:2379] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 8: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:13.198107 13194 raft_consensus.cc:686] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 10 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:13.201077 13046 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.201004 12887 consensus_queue.cc:1046] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.005s
I20240426 09:56:13.204077 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.206264 10081 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:56:13.208343 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.209070 9935 raft_consensus.cc:2354] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 8. Current term is 10.
I20240426 09:56:13.213481 9932 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 13.
W20240426 09:56:13.214274 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 52007.1us
I20240426 09:56:13.199314 13128 raft_consensus.cc:3032] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Stepping down as leader of term 9
I20240426 09:56:13.216077 13193 consensus_queue.cc:1035] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:56:13.216573 13129 consensus_queue.cc:1046] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20240426 09:56:13.218122 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.220475 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc5aa33dc824544ade18161bf2800a6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.157089 13196 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:13.204576 13152 raft_consensus.cc:2726] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 09:56:13.208643 13136 raft_consensus.cc:686] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [term 6 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:56:13.222930 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:13.245569 9938 raft_consensus.cc:2379] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 9: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:13.223232 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.223850 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.224804 13148 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.225368 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.189047 13146 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:13.225804 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.247303 9935 raft_consensus.cc:2354] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6.
I20240426 09:56:13.247443 9941 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:56:13.226068 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.248733 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.229418 13123 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.230322 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.200265 12976 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.199451 12983 consensus_queue.cc:1046] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20240426 09:56:13.232293 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.226531 13117 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.233906 9906 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:13.236079 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.252665 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:56:13.237660 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 51580.6us
I20240426 09:56:13.240212 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.242230 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.246548 10082 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:13.234570 13128 raft_consensus.cc:729] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:13.251929 10088 raft_consensus.cc:2354] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8.
I20240426 09:56:13.255870 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.255604 13128 consensus_queue.cc:260] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:13.256495 10085 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:56:13.257117 13128 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:13.255659 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.255900 9932 raft_consensus.cc:2354] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8.
I20240426 09:56:13.262106 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.263536 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.263886 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.264611 10081 raft_consensus.cc:2354] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6.
I20240426 09:56:13.252851 9939 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:56:13.265292 13136 consensus_queue.cc:237] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:13.266598 12976 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:13.266793 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.267072 9906 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:13.267395 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.263463 13046 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:13.265404 12983 raft_consensus.cc:3032] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:56:13.265415 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.267891 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.264535 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.263129 10082 raft_consensus.cc:2396] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20240426 09:56:13.263193 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.270951 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.272851 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:13.274176 13152 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 09:56:13.274513 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.275352 10090 raft_consensus.cc:2379] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 8: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:13.274581 13152 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:13.276811 13152 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:13.277916 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.282553 10087 raft_consensus.cc:2354] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6.
I20240426 09:56:13.295475 9934 raft_consensus.cc:2379] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 10: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:13.279243 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.300565 10083 raft_consensus.cc:2354] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 10. Current term is 11.
I20240426 09:56:13.267947 10084 raft_consensus.cc:2454] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:56:13.305379 9941 raft_consensus.cc:1226] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 8. Current term is 9. Ops: []
I20240426 09:56:13.214922 13145 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:13.212018 12969 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.268478 12983 raft_consensus.cc:729] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:13.251627 13204 raft_consensus.cc:3032] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:56:13.331037 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.336910 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.331681 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.337684 9937 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 9.
I20240426 09:56:13.338799 9940 raft_consensus.cc:2354] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 9. Current term is 10.
W20240426 09:56:13.341271 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 88212.9us
I20240426 09:56:13.342111 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.343326 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.344259 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.345285 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:13.345414 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.345868 9939 raft_consensus.cc:2362] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8. Re-sending same reply.
I20240426 09:56:13.346668 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.346876 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.347599 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.347748 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.347412 12983 consensus_queue.cc:260] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:13.347903 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.348335 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.348531 10087 raft_consensus.cc:2454] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:56:13.346431 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.330966 12903 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:13.350693 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:13.351275 9940 raft_consensus.cc:2362] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6. Re-sending same reply.
I20240426 09:56:13.349406 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.352478 10084 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:56:13.353235 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.353834 10084 raft_consensus.cc:2379] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 5: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:13.354175 12985 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.354801 9936 raft_consensus.cc:1226] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 3. Current term is 4. Ops: []
I20240426 09:56:13.264683 13196 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:13.393843 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.395334 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.395160 12969 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:13.395896 10089 raft_consensus.cc:2354] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8.
I20240426 09:56:13.395987 10081 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:56:13.349311 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.395071 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.355144 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:56:13.398564 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 96797.5us
W20240426 09:56:13.399044 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 59305.6us
I20240426 09:56:13.355365 9934 raft_consensus.cc:2396] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20240426 09:56:13.348934 9938 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:56:13.400449 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.401247 10586 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:13.362455 10082 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 12 FOLLOWER]: Advancing to term 13
W20240426 09:56:13.385599 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 43674.3us
I20240426 09:56:13.354465 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.398850 10085 raft_consensus.cc:2354] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8.
I20240426 09:56:13.348771 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:13.331225 13194 consensus_queue.cc:237] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.407369 10616 raft_consensus.cc:1226] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:13.408187 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.408394 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.408784 10081 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 9.
I20240426 09:56:13.409405 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.410212 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.351233 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:13.412760 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.324342 12976 raft_consensus.cc:2726] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 09:56:13.328220 13146 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 11 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:13.348815 12983 raft_consensus.cc:3037] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:13.413586 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.421733 9906 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:56:13.338855 13204 raft_consensus.cc:729] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:13.346555 13152 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.354538 13145 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:13.423313 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.422307 10084 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:56:13.427078 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.434917 10081 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:56:13.428674 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.431375 10087 raft_consensus.cc:2379] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 9: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:13.410607 13058 raft_consensus.cc:2781] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 09:56:13.442746 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.444692 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.445380 9906 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:13.442829 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.434286 10089 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:56:13.454918 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.455271 9939 raft_consensus.cc:1226] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 5. Current term is 6. Ops: []
I20240426 09:56:13.459865 10081 raft_consensus.cc:2454] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:56:13.458799 10087 raft_consensus.cc:1261] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20240426 09:56:13.455927 10088 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:56:13.455869 10083 raft_consensus.cc:1226] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 6. Current term is 7. Ops: []
I20240426 09:56:13.456394 9941 raft_consensus.cc:1226] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 9. Current term is 10. Ops: []
I20240426 09:56:13.337779 12963 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.458721 9932 raft_consensus.cc:1226] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 9. Current term is 10. Ops: []
I20240426 09:56:13.460866 9906 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:13.443591 12990 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 09:56:13.467823 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.463588 13149 raft_consensus.cc:2781] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:13.468000 9941 raft_consensus.cc:1226] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 3. Current term is 4. Ops: []
I20240426 09:56:13.465847 9937 raft_consensus.cc:1261] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 09:56:13.442965 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:13.469472 9906 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:13.469741 10085 raft_consensus.cc:2379] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 6: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:13.470407 10089 raft_consensus.cc:1226] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 4. Current term is 5. Ops: []
I20240426 09:56:13.469038 13149 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:13.473132 13149 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:13.475531 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:13.476233 9938 raft_consensus.cc:2379] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 8: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:13.477124 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.471655 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.479161 10090 raft_consensus.cc:2379] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 6: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:13.479001 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:13.480515 9937 raft_consensus.cc:1226] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20240426 09:56:13.426604 12969 consensus_queue.cc:1046] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20240426 09:56:13.499372 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:13.500255 9941 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:13.422644 13014 consensus_queue.cc:1046] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20240426 09:56:13.501737 9933 raft_consensus.cc:2379] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 5: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:13.460081 13058 raft_consensus.cc:686] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [term 9 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:56:13.509460 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:13.505870 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.502306 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.513916 10083 raft_consensus.cc:2354] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 6. Current term is 7.
I20240426 09:56:13.509691 10082 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 13.
I20240426 09:56:13.512610 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:13.515159 10012 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:13.476123 12985 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:13.521270 10012 raft_consensus.cc:2454] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:13.467052 13117 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:13.525369 9906 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:13.559717 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:13.560353 10014 raft_consensus.cc:3037] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:13.535243 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.535305 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.536273 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:13.562531 10010 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 8
I20240426 09:56:13.536671 10013 raft_consensus.cc:1226] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20240426 09:56:13.558360 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:13.539743 13149 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.558280 9936 raft_consensus.cc:1226] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 4. Current term is 5. Ops: []
I20240426 09:56:13.565330 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.566006 10082 raft_consensus.cc:2354] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 8. Current term is 9.
I20240426 09:56:13.567751 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:13.568003 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:13.568310 10009 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:13.568708 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:13.569873 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:13.570379 10012 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:13.571770 9937 raft_consensus.cc:1226] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 6. Current term is 7. Ops: []
I20240426 09:56:13.572751 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:13.573789 9940 raft_consensus.cc:1226] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 6. Current term is 7. Ops: []
I20240426 09:56:13.499058 12976 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20240426 09:56:13.576974 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 51702us
I20240426 09:56:13.578495 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.580001 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.580760 10081 raft_consensus.cc:2379] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 6: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:13.577080 9982 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:13.579067 10087 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 10.
I20240426 09:56:13.583120 9941 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 11.
I20240426 09:56:13.577041 10009 raft_consensus.cc:2454] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 10.
I20240426 09:56:13.582584 10056 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:13.589066 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:13.511714 13148 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:13.642190 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 59668.3us
I20240426 09:56:13.657662 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:13.657842 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.746449 10011 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:13.541805 13204 consensus_queue.cc:260] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:13.595005 9933 raft_consensus.cc:1226] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 8. Current term is 9. Ops: []
I20240426 09:56:13.735781 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.523037 13202 raft_consensus.cc:2726] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:13.739980 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.735225 10012 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:13.757683 10087 raft_consensus.cc:2354] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for earlier term 12. Current term is 13.
W20240426 09:56:13.734961 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 93890.1us
I20240426 09:56:13.741480 10082 raft_consensus.cc:1226] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 8. Current term is 9. Ops: []
I20240426 09:56:13.741987 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.736707 10014 raft_consensus.cc:2454] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
W20240426 09:56:13.744354 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 199143us
W20240426 09:56:13.739923 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 157207us
I20240426 09:56:13.759697 9906 leader_election.cc:302] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:13.595690 13123 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:13.751142 9982 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:13.586942 12976 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:13.774649 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.775267 13152 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:13.775903 13210 log.cc:924] Time spent T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.307s user 0.001s sys 0.000s
I20240426 09:56:13.767079 9906 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:13.739311 13117 raft_consensus.cc:2781] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:56:13.749903 13119 raft_consensus.cc:2781] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election won for term 7
W20240426 09:56:13.765233 12987 consensus_meta.cc:287] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.591s user 0.007s sys 0.003s
I20240426 09:56:13.797916 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.763900 13194 raft_consensus.cc:2781] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:13.793175 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.801986 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.802883 10082 raft_consensus.cc:2362] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 7. Re-sending same reply.
I20240426 09:56:13.805512 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.764168 12969 raft_consensus.cc:2781] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election won for term 10
W20240426 09:56:13.770395 13129 consensus_queue.cc:1159] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "826f03b2fb9c4b9483623b041a1bf80e" responder_term: 7 status { last_received { term: 7 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 7 index: 7 } }
I20240426 09:56:13.796617 13152 raft_consensus.cc:2781] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:56:13.788513 13204 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:13.786406 12976 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:13.801581 12987 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.820777 12987 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:13.821188 13194 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:13.821586 13194 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:13.792361 12917 raft_consensus.cc:2781] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:13.823063 12917 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:13.823397 12917 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:13.823796 10086 raft_consensus.cc:2379] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 8: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:13.827389 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.828747 10056 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:13.790586 9906 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:13.831454 9938 raft_consensus.cc:2354] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 6. Current term is 7.
I20240426 09:56:13.795894 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:56:13.807579 13214 log.cc:924] Time spent T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.167s user 0.001s sys 0.000s
I20240426 09:56:13.811265 9933 raft_consensus.cc:1261] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 09:56:13.810849 13058 consensus_queue.cc:237] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:13.846201 10087 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 10.
I20240426 09:56:13.800674 12990 raft_consensus.cc:686] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 10 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:13.795089 13119 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:13.867022 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.814862 12963 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:13.877487 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 48805.5us
I20240426 09:56:13.892802 10056 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:13.893549 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.878141 9839 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50252 (request call id 330) took 7229 ms. Trace:
I20240426 09:56:13.885704 10011 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 10.
I20240426 09:56:13.894505 10056 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:13.885251 12940 raft_consensus.cc:3032] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:56:13.907049 12940 raft_consensus.cc:729] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
W20240426 09:56:13.883244 13211 log.cc:924] Time spent T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.411s user 0.003s sys 0.000s
W20240426 09:56:13.876942 13217 log.cc:924] Time spent T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.405s user 0.002s sys 0.000s
I20240426 09:56:13.915308 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.918027 10084 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 7.
I20240426 09:56:13.877699 13207 raft_consensus.cc:2781] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:56:13.881620 13119 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:13.881450 13149 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:13.922250 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:13.907209 13128 raft_consensus.cc:2726] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240426 09:56:13.928512 10583 leader_election.cc:334] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 1.798s (SENT)
I20240426 09:56:13.930629 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.937064 10010 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 8.
W20240426 09:56:13.940593 13049 consensus_meta.cc:287] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.756s user 0.009s sys 0.000s
I20240426 09:56:13.919566 13146 raft_consensus.cc:2726] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 09:56:13.894948 9839 rpcz_store.cc:276] 0426 09:56:06.648698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:06.648851 (+ 153us) service_pool.cc:225] Handling call
0426 09:56:07.594531 (+945680us) spinlock_profiling.cc:243] Waited 312 ms on lock 0x7b54000041d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcbc70 00007f9ca2c4f0ba 00007f9ca8ba096c 00007f9ca8c9a8de 00007f9ca6faea04 00007f9ca6fae992 00007f9ca6fae8c1 00007f9ca6fae83c 00007f9ca6fadaf2 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:13.878049 (+6283518us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 09:56:12.683275 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 09:56:12.683861 (+ 586us) write_op.cc:424] Acquiring schema lock in shared mode
0426 09:56:12.683909 (+ 48us) write_op.cc:427] Acquired schema lock
0426 09:56:12.683921 (+ 12us) tablet.cc:660] Decoding operations
0426 09:56:12.684300 (+ 379us) write_op.cc:612] Acquiring the partition lock for write op
0426 09:56:12.684370 (+ 70us) write_op.cc:633] Partition lock acquired for write op
0426 09:56:12.684396 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
0426 09:56:12.684670 (+ 274us) tablet.cc:699] Row locks acquired
0426 09:56:12.684696 (+ 26us) write_op.cc:260] PREPARE: finished
0426 09:56:12.684847 (+ 151us) write_op.cc:270] Start()
0426 09:56:12.684974 (+ 127us) write_op.cc:276] Timestamp: P: 1714125372684785 usec, L: 0
0426 09:56:12.685002 (+ 28us) op_driver.cc:331] REPLICATION: starting
0426 09:56:12.685625 (+ 623us) log.cc:842] Serialized 541 byte log entry
0426 09:56:13.399105 (+713480us) op_driver.cc:444] REPLICATION: finished
0426 09:56:13.815034 (+415929us) write_op.cc:301] APPLY: starting
0426 09:56:13.815136 (+ 102us) tablet.cc:1365] starting BulkCheckPresence
0426 09:56:13.815262 (+ 126us) tablet.cc:1368] finished BulkCheckPresence
0426 09:56:13.815373 (+ 111us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 09:56:13.815671 (+ 298us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 09:56:13.815911 (+ 240us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0426 09:56:13.815965 (+ 54us) write_op.cc:312] APPLY: finished
0426 09:56:13.816227 (+ 262us) log.cc:842] Serialized 23 byte log entry
0426 09:56:13.816381 (+ 154us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:56:13.816461 (+ 80us) write_op.cc:446] Released schema lock
0426 09:56:13.816510 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0426 09:56:13.816802 (+ 292us) write_op.cc:481] Releasing partition, row and schema locks
0426 09:56:13.816828 (+ 26us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":4674431,"reported_tablets":5,"spinlock_wait_cycles":718107008,"child_traces":[["op",{"apply.queue_time_us":392001,"apply.run_cpu_time_us":2047,"apply.run_wall_time_us":2047,"num_ops":1,"prepare.queue_time_us":170686,"prepare.run_cpu_time_us":63980,"prepare.run_wall_time_us":256305,"raft.queue_time_us":267957,"raft.run_cpu_time_us":14630,"raft.run_wall_time_us":289650,"replication_time_us":679458,"thread_start_us":636863,"threads_started":4,"wal-append.queue_time_us":309951}]]}
I20240426 09:56:13.926007 13207 raft_consensus.cc:686] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:13.944382 9939 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 9.
W20240426 09:56:13.952955 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 58508.9us
I20240426 09:56:13.918843 13014 consensus_queue.cc:1035] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
W20240426 09:56:13.962638 12983 consensus_meta.cc:287] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.532s user 0.007s sys 0.000s
I20240426 09:56:13.969281 12917 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.959211 12940 consensus_queue.cc:260] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:13.971735 12940 raft_consensus.cc:3037] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:56:13.971462 13207 consensus_queue.cc:237] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:13.943842 12990 consensus_queue.cc:237] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:13.978993 10088 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 6.
I20240426 09:56:13.982751 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:13.983275 10089 raft_consensus.cc:1226] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 4. Current term is 5. Ops: []
I20240426 09:56:13.984484 10086 raft_consensus.cc:1261] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 09:56:13.985658 10082 raft_consensus.cc:1226] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 8. Current term is 9. Ops: []
I20240426 09:56:14.001742 9905 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:14.002912 13194 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.004091 10083 raft_consensus.cc:2379] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 9: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:14.004899 13194 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:14.006260 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:14.006947 13203 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:14.007498 13203 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.009389 13203 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:14.010342 12969 raft_consensus.cc:686] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 10 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:14.010883 13145 raft_consensus.cc:2726] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:56:14.011328 12969 consensus_queue.cc:237] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:14.012212 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 221701us
I20240426 09:56:14.014286 9906 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:14.014952 12985 raft_consensus.cc:2781] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:56:14.015458 12985 raft_consensus.cc:686] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [term 6 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:14.015714 12976 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.017633 12976 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:14.017928 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:14.018524 10012 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:56:14.019480 10081 raft_consensus.cc:1261] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 09:56:14.018514 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:14.020239 10011 raft_consensus.cc:3037] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:14.021019 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:14.021602 10010 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:14.021862 10056 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:14.019554 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:56:14.021874 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 270805us
I20240426 09:56:14.023444 10014 raft_consensus.cc:3037] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 9
I20240426 09:56:14.019037 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:14.028375 10008 raft_consensus.cc:2379] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 7: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:56:14.028909 10007 raft_consensus.cc:1261] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20240426 09:56:14.030965 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 74581.2us
I20240426 09:56:14.031473 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:14.031826 10056 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:14.032414 10015 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:14.037596 10009 raft_consensus.cc:1226] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 6. Current term is 7. Ops: []
I20240426 09:56:14.041774 10008 raft_consensus.cc:1226] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240426 09:56:14.038648 9982 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:14.043888 10056 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:14.050822 13146 raft_consensus.cc:2726] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:56:14.041085 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:14.055687 10013 raft_consensus.cc:2379] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 7: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:14.066069 10015 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:14.067312 10014 raft_consensus.cc:2454] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 9.
I20240426 09:56:14.069284 9982 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:14.037009 12983 raft_consensus.cc:2781] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election won for term 6
W20240426 09:56:14.074393 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 47774us
I20240426 09:56:14.054461 10084 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 6.
W20240426 09:56:14.073921 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 59710.6us
I20240426 09:56:14.087342 9906 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:14.100589 10010 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 10.
I20240426 09:56:14.102555 9906 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:14.105151 9904 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:14.106272 9904 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:14.032295 13119 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:14.102613 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 58777.8us
I20240426 09:56:14.054791 13194 raft_consensus.cc:2781] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:14.114337 10011 raft_consensus.cc:2454] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:56:14.122854 12985 consensus_queue.cc:237] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.123605 13227 raft_consensus.cc:2726] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:56:14.114547 13194 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:14.101066 12917 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:14.127365 10013 raft_consensus.cc:1261] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 09:56:14.129462 13128 consensus_queue.cc:1046] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:56:14.130030 13146 consensus_queue.cc:1046] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 09:56:14.131847 13119 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:14.097966 13123 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 09:56:14.114758 12983 raft_consensus.cc:686] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:14.112875 13049 raft_consensus.cc:2781] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:56:14.124331 13227 raft_consensus.cc:2726] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240426 09:56:14.195487 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 89296.1us
I20240426 09:56:14.151829 9906 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:14.194555 13203 raft_consensus.cc:2781] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:14.118935 13225 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 09:56:14.204478 9904 leader_election.cc:398] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 9. Current term is 10.
I20240426 09:56:14.204974 9904 leader_election.cc:401] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20240426 09:56:14.191452 9839 catalog_manager.cc:5484] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 8 to 9, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 9 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:56:14.138084 13194 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:14.152107 13148 raft_consensus.cc:2781] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20240426 09:56:14.214814 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 65300.4us
I20240426 09:56:14.138303 13046 consensus_queue.cc:1035] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20240426 09:56:14.149300 13160 consensus_queue.cc:1046] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20240426 09:56:14.136880 13196 consensus_queue.cc:1046] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20240426 09:56:14.203094 13123 raft_consensus.cc:686] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 10 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:14.197227 13152 raft_consensus.cc:686] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:14.201656 13117 raft_consensus.cc:686] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:14.217576 9839 catalog_manager.cc:5484] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 5 to 6, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 6 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:56:14.204277 13203 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:14.205438 13225 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:14.229414 13149 raft_consensus.cc:2726] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 9. Current term is 10.
I20240426 09:56:14.228502 12969 raft_consensus.cc:2781] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 09:56:14.231128 13207 consensus_queue.cc:1046] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 09:56:14.232388 13152 consensus_queue.cc:237] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.234841 13203 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:14.244545 13203 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.248402 12969 raft_consensus.cc:686] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [term 9 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:14.249301 12969 consensus_queue.cc:237] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.250866 13203 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:14.253113 13046 raft_consensus.cc:3032] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Stepping down as leader of term 3
I20240426 09:56:14.253536 13046 raft_consensus.cc:729] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:14.258221 13058 consensus_queue.cc:1035] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:56:14.214114 13049 raft_consensus.cc:686] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [term 8 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:14.259079 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fe10a2b3746f9909d3596580b6854" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:14.258630 10010 raft_consensus.cc:1226] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 5. Current term is 6. Ops: []
W20240426 09:56:14.259292 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 166638us
I20240426 09:56:14.216293 13148 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:14.276558 10014 raft_consensus.cc:1226] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 4. Current term is 5. Ops: []
W20240426 09:56:14.276964 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 207728us
W20240426 09:56:14.263545 9979 leader_election.cc:334] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 1.515s (SENT)
I20240426 09:56:14.278144 9982 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:56:14.271925 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:14.272748 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:14.283751 10007 raft_consensus.cc:2379] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 8: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:14.284499 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:14.285161 10007 raft_consensus.cc:2354] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7.
I20240426 09:56:14.260708 9906 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:14.260514 9904 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:56:14.229707 9839 catalog_manager.cc:5484] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 9 to 10, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to f12521b6d51242629cf4297864df00b9 (127.8.141.129). New cstate: current_term: 10 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:56:14.206697 12976 consensus_queue.cc:1035] T cf8634deb0a5425db9c47915d49386d5 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20240426 09:56:14.287684 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:14.289367 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:14.305121 10009 raft_consensus.cc:3037] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:14.289510 10016 raft_consensus.cc:1261] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 09:56:14.289970 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:14.306787 10008 raft_consensus.cc:2379] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 9: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:56:14.290809 10011 raft_consensus.cc:1226] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 9. Current term is 10. Ops: []
I20240426 09:56:14.295022 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:14.295727 12950 consensus_queue.cc:1035] T 1c0134eb2ced4f5c8e56412aaee1c225 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:56:14.282727 13148 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:14.297744 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:14.259306 13225 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:14.299536 10010 raft_consensus.cc:1261] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240426 09:56:14.300642 9906 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:14.304141 9904 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:14.288460 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:14.316634 10007 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:14.302553 9839 catalog_manager.cc:5484] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 7 to 9, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 9 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:56:14.317175 9982 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:14.317781 9839 catalog_manager.cc:5484] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 7 to 9, leader changed from 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 9 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:56:14.319658 9839 catalog_manager.cc:5484] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:56:14.326560 9982 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:14.254307 13046 consensus_queue.cc:260] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.330415 13123 consensus_queue.cc:237] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.336118 13119 raft_consensus.cc:2726] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 09:56:14.336894 13223 raft_consensus.cc:2781] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 09:56:14.337270 9982 leader_election.cc:302] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
W20240426 09:56:14.383041 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 45849.7us
I20240426 09:56:14.383718 9982 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:14.385174 9982 leader_election.cc:398] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6.
I20240426 09:56:14.385426 9982 leader_election.cc:401] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20240426 09:56:14.385984 9982 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:14.339159 12983 consensus_queue.cc:237] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.346498 9979 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:56:14.328109 13227 raft_consensus.cc:2781] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:14.351483 13194 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.363596 10007 raft_consensus.cc:2454] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 10.
I20240426 09:56:14.314828 13129 raft_consensus.cc:2781] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:14.301198 13237 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 09:56:14.365512 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
W20240426 09:56:14.393970 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 47548.5us
I20240426 09:56:14.394964 9979 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:14.395772 10015 raft_consensus.cc:2379] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 9: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:14.370416 12950 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
W20240426 09:56:14.387142 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 83059us
I20240426 09:56:14.386829 13119 raft_consensus.cc:2726] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240426 09:56:14.396875 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 199040us
I20240426 09:56:14.388263 10009 raft_consensus.cc:2454] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 11.
I20240426 09:56:14.398340 9904 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:56:14.347162 13117 consensus_queue.cc:237] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:14.390648 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 267735us
I20240426 09:56:14.389986 13149 raft_consensus.cc:2726] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:14.389746 12917 raft_consensus.cc:2726] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:14.388849 13223 raft_consensus.cc:686] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:14.390439 13227 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:14.400310 9839 catalog_manager.cc:5484] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 8 to 9, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 9 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:56:14.409703 9839 catalog_manager.cc:5484] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 7 to 9, leader changed from f12521b6d51242629cf4297864df00b9 (127.8.141.129) to d3a480e831d649139ff016fb68aee3ee (127.8.141.131). New cstate: current_term: 9 leader_uuid: "d3a480e831d649139ff016fb68aee3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } health_report { overall_health: HEALTHY } } }
I20240426 09:56:14.392796 13233 raft_consensus.cc:2741] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20240426 09:56:14.395056 13129 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
W20240426 09:56:14.406491 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 277299us
I20240426 09:56:14.410825 9904 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:56:14.415393 9904 leader_election.cc:398] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 10. Current term is 11.
I20240426 09:56:14.415701 9904 leader_election.cc:401] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20240426 09:56:14.395627 13046 raft_consensus.cc:3037] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:56:14.397825 13237 raft_consensus.cc:686] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [term 10 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:14.419606 10084 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:56:14.424531 10084 raft_consensus.cc:595] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:14.424906 10084 raft_consensus.cc:603] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 09:56:14.420555 13194 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:14.422544 13128 raft_consensus.cc:3032] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Stepping down as leader of term 9
I20240426 09:56:14.396657 13240 raft_consensus.cc:2726] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6.
I20240426 09:56:14.437047 9904 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68
W20240426 09:56:14.464205 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 45680.7us
I20240426 09:56:14.412287 13227 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:14.383667 13049 consensus_queue.cc:237] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.465309 13128 raft_consensus.cc:729] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:14.429167 9839 catalog_manager.cc:5484] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 5 to 6, leader changed from d3a480e831d649139ff016fb68aee3ee (127.8.141.131) to 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130). New cstate: current_term: 6 leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } }
I20240426 09:56:14.415166 13129 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:14.465126 9904 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:14.417013 13119 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 13 FOLLOWER]: Leader election won for term 13
I20240426 09:56:14.412899 13152 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 09:56:14.471980 10055 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
W20240426 09:56:14.477619 10583 leader_election.cc:334] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 2.138s (SENT)
I20240426 09:56:14.476619 13207 raft_consensus.cc:3032] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Stepping down as leader of term 9
I20240426 09:56:14.483108 13207 raft_consensus.cc:729] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:14.483839 13207 consensus_queue.cc:260] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.483975 9839 catalog_manager.cc:5484] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "f12521b6d51242629cf4297864df00b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } } }
I20240426 09:56:14.484861 13227 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:14.492656 13227 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:14.495083 13160 raft_consensus.cc:3032] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:56:14.495524 13160 raft_consensus.cc:729] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:14.496333 13160 consensus_queue.cc:260] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.497507 13160 raft_consensus.cc:3037] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:14.403399 12950 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:14.500404 12950 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:14.501343 13196 raft_consensus.cc:3032] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Stepping down as leader of term 9
I20240426 09:56:14.501727 13196 raft_consensus.cc:729] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:14.502525 13196 consensus_queue.cc:260] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.503986 13196 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:14.510594 12987 raft_consensus.cc:2726] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 09:56:14.511101 13119 raft_consensus.cc:686] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [term 13 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:14.512053 13119 consensus_queue.cc:237] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.12, Last appended by leader: 12, Current term: 13, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:14.439199 12976 raft_consensus.cc:2726] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 10. Current term is 11.
I20240426 09:56:14.518911 13245 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:14.515697 13225 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.520184 13245 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:14.484972 13207 raft_consensus.cc:3037] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:14.522112 13245 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:14.523212 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:14.523403 13225 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:14.523833 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:14.524545 9933 raft_consensus.cc:2379] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 9: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:14.522003 13122 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:14.529356 13122 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:14.531818 13122 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:14.532630 13237 consensus_queue.cc:237] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.535645 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 14 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:14.423419 12917 raft_consensus.cc:2726] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20240426 09:56:14.510746 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 45716.2us
I20240426 09:56:14.546854 9904 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:14.439000 13236 raft_consensus.cc:2781] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:14.544418 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 14 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:14.536306 9933 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 13.
I20240426 09:56:14.471216 12969 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 09:56:14.561345 9904 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:14.558894 13145 raft_consensus.cc:2726] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20240426 09:56:14.574087 13249 log.cc:924] Time spent T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20240426 09:56:14.580202 9933 raft_consensus.cc:1261] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 09:56:14.581856 13136 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240426 09:56:14.584842 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 112924us
I20240426 09:56:14.585598 10055 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:14.555262 13236 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:14.558523 13152 raft_consensus.cc:686] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [term 9 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:14.562188 12969 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:14.564126 12950 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:56:14.598134 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 203230us
I20240426 09:56:14.598243 10055 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:14.591779 9904 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:14.599918 9904 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:14.589890 13236 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:14.517160 13223 consensus_queue.cc:237] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.593477 13160 raft_consensus.cc:2726] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20240426 09:56:14.606293 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 128700us
I20240426 09:56:14.594063 13225 raft_consensus.cc:2726] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240426 09:56:14.620297 10583 leader_election.cc:334] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 2.191s (SENT)
I20240426 09:56:14.596966 12969 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:14.600577 9979 leader_election.cc:398] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6.
I20240426 09:56:14.613893 13244 raft_consensus.cc:2726] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:14.605850 13203 raft_consensus.cc:2726] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 09:56:14.613745 13248 raft_consensus.cc:2726] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:56:14.599143 13239 raft_consensus.cc:2781] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:14.600957 12976 raft_consensus.cc:2726] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 09:56:14.637348 13229 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:14.602990 12950 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:14.624119 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:14.594462 13128 consensus_queue.cc:260] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:14.664729 13234 log.cc:924] Time spent T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
I20240426 09:56:14.670444 13239 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:14.670985 13239 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:14.672780 10615 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20240426 09:56:14.651593 13152 consensus_queue.cc:237] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:14.686833 13257 raft_consensus.cc:482] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:14.687433 13257 raft_consensus.cc:504] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.691730 13229 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:14.694861 12976 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240426 09:56:14.774225 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 174366us
W20240426 09:56:14.774421 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 154150us
I20240426 09:56:14.776875 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:56:14.605991 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 220051us
I20240426 09:56:14.778198 9982 leader_election.cc:398] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 8. Current term is 10.
I20240426 09:56:14.778870 9982 leader_election.cc:401] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20240426 09:56:14.782387 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:14.786654 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:56:14.788646 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 188184us
I20240426 09:56:14.789783 9979 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:14.790861 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:14.792143 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e11de7e65e544ffac5f7683e9614536" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:14.792989 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:14.793695 10082 raft_consensus.cc:2379] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 9: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:14.793946 13129 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:14.795033 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:14.795383 13229 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:14.796613 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:56:14.775135 10583 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 1.573s (SENT) [suppressed 69 similar messages]
I20240426 09:56:14.798193 10084 raft_consensus.cc:2396] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20240426 09:56:14.800897 10583 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:14.802063 13192 raft_consensus.cc:2726] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:14.802906 12991 tablet_copy_client.cc:523] T 1b76245317c24e129e7f16cdb4d99007 P 1fee3da01ebc463396397057fdb81a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 09:56:14.813676 10009 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:14.807159 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:14.807533 12950 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:14.807724 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:14.808240 13259 consensus_queue.cc:1046] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:56:14.815002 10088 raft_consensus.cc:3037] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:14.818358 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:14.818969 10089 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:14.820225 10087 raft_consensus.cc:1261] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
I20240426 09:56:14.820569 10087 pending_rounds.cc:76] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee: Aborting all ops after (but not including) 4
I20240426 09:56:14.822229 10086 raft_consensus.cc:1261] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:56:14.814733 9935 raft_consensus.cc:2454] T be50ac00c79946d89ecaf230084a401d P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:56:14.808095 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:56:14.825951 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 227795us
I20240426 09:56:14.820770 10087 pending_rounds.cc:97] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20240426 09:56:14.828291 10087 raft_consensus.cc:2866] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:56:14.809686 13148 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.806980 13128 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:14.829768 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:14.833654 10009 raft_consensus.cc:1261] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 09:56:14.834365 9939 raft_consensus.cc:1226] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 6. Current term is 7. Ops: []
I20240426 09:56:14.831187 9936 raft_consensus.cc:1226] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20240426 09:56:14.832831 9937 raft_consensus.cc:1261] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20240426 09:56:14.835568 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 45871us
I20240426 09:56:14.836311 9979 leader_election.cc:302] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:14.837792 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:14.838400 10009 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 10.
I20240426 09:56:14.842242 10090 raft_consensus.cc:1261] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 09:56:14.850833 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:14.851595 10084 raft_consensus.cc:2354] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7.
I20240426 09:56:14.861639 9904 leader_election.cc:398] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7.
I20240426 09:56:14.864838 9904 leader_election.cc:401] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20240426 09:56:14.873361 9979 leader_election.cc:398] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6.
I20240426 09:56:14.874305 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:14.874994 9940 raft_consensus.cc:3037] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:14.883071 9941 raft_consensus.cc:1226] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 8. Current term is 9. Ops: []
W20240426 09:56:14.883344 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 58156.6us
I20240426 09:56:14.884032 10056 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:14.886080 9932 raft_consensus.cc:1226] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 6. Current term is 7. Ops: []
I20240426 09:56:14.839927 13149 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:14.897478 9979 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:14.898268 10055 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:14.894542 10056 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:14.895092 13257 leader_election.cc:288] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:14.843582 12917 raft_consensus.cc:2726] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 09:56:14.907233 13129 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:14.903453 13248 consensus_queue.cc:1046] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20240426 09:56:14.903817 13149 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:14.910717 13149 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:14.911718 10089 raft_consensus.cc:2454] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 10.
I20240426 09:56:14.912312 12969 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:14.913478 13259 raft_consensus.cc:2726] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 09:56:14.883054 13049 consensus_queue.cc:1035] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.031s
I20240426 09:56:14.915424 10056 leader_election.cc:398] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 6. Current term is 7.
I20240426 09:56:14.915740 10056 leader_election.cc:401] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20240426 09:56:14.833992 13207 consensus_queue.cc:1046] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20240426 09:56:14.841965 13202 consensus_queue.cc:1035] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20240426 09:56:14.916429 13242 consensus_queue.cc:1046] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 09:56:14.887815 13239 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:14.872340 13226 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:14.918078 10055 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:14.919111 10056 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:14.921670 9904 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:14.922698 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:14.922888 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:14.944712 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:14.945109 10087 raft_consensus.cc:2379] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 10: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:56:14.931382 13196 raft_consensus.cc:3032] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:56:14.923259 13236 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:14.932962 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 154881us
I20240426 09:56:14.882366 13203 consensus_queue.cc:1035] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20240426 09:56:14.951296 9982 leader_election.cc:302] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:14.935063 13227 raft_consensus.cc:2726] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 8. Current term is 10.
I20240426 09:56:14.935176 13119 raft_consensus.cc:2726] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 09:56:14.935846 9940 raft_consensus.cc:2454] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
W20240426 09:56:14.900662 13252 log.cc:924] Time spent T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240426 09:56:14.914170 12969 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:14.888407 13233 raft_consensus.cc:2726] T 5c0fe10a2b3746f9909d3596580b6854 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:56:14.992239 9934 raft_consensus.cc:1226] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 7. Current term is 8. Ops: []
W20240426 09:56:14.943689 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 95107.8us
I20240426 09:56:14.945785 10088 raft_consensus.cc:2454] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:14.933836 13148 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:14.934579 9979 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
W20240426 09:56:14.969281 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 47683.3us
I20240426 09:56:14.950060 13196 raft_consensus.cc:729] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:14.903970 13194 raft_consensus.cc:2781] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Leader election won for term 11
W20240426 09:56:14.984336 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 140126us
I20240426 09:56:14.939496 12985 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:56:14.985648 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 67737.1us
W20240426 09:56:14.986155 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 67127.4us
I20240426 09:56:14.914186 13255 raft_consensus.cc:2726] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:14.923238 13229 consensus_queue.cc:1046] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.045s
I20240426 09:56:14.990410 10013 raft_consensus.cc:1226] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:14.924590 13058 consensus_queue.cc:1035] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20240426 09:56:14.882236 13230 consensus_queue.cc:1046] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20240426 09:56:15.004943 10585 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:15.005869 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:14.989300 13240 raft_consensus.cc:2726] T ddc5aa33dc824544ade18161bf2800a6 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:15.011212 10055 leader_election.cc:302] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:14.985172 13145 raft_consensus.cc:2781] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election won for term 10
W20240426 09:56:15.013710 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 166245us
I20240426 09:56:15.000077 13146 raft_consensus.cc:2726] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee for earlier term 6. Current term is 7.
W20240426 09:56:15.020094 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 68868.5us
I20240426 09:56:14.991974 13160 raft_consensus.cc:3032] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:56:15.034945 9982 leader_election.cc:398] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6.
I20240426 09:56:15.035257 9982 leader_election.cc:401] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20240426 09:56:15.000923 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 66402.8us
I20240426 09:56:15.036478 9979 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:15.037600 9979 leader_election.cc:398] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8.
I20240426 09:56:15.037912 9979 leader_election.cc:401] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20240426 09:56:14.991740 12983 raft_consensus.cc:3032] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:56:14.998878 13244 raft_consensus.cc:2726] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:56:14.999624 13259 raft_consensus.cc:2741] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20240426 09:56:15.011180 13227 raft_consensus.cc:2726] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 09:56:14.986954 13117 raft_consensus.cc:2726] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7.
I20240426 09:56:15.038944 9904 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:14.983311 13226 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:15.023268 13194 raft_consensus.cc:686] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 11 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:15.046533 9979 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:56:15.016772 13242 consensus_queue.cc:1035] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20240426 09:56:15.034816 13160 raft_consensus.cc:729] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:15.053352 10089 raft_consensus.cc:1261] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20240426 09:56:15.091627 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 79365.8us
W20240426 09:56:15.097493 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 80828.8us
I20240426 09:56:15.016595 12985 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:15.127463 13263 log.cc:924] Time spent T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20240426 09:56:15.044677 13229 raft_consensus.cc:3032] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:56:15.040445 12983 raft_consensus.cc:729] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:15.041976 13240 raft_consensus.cc:3037] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:15.047461 13129 raft_consensus.cc:2781] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:56:15.047674 13233 raft_consensus.cc:2726] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6.
I20240426 09:56:15.037180 13145 raft_consensus.cc:686] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 10 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:15.133101 13239 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:15.134913 13226 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:15.050704 13196 consensus_queue.cc:260] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:15.135699 13244 raft_consensus.cc:2726] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 09:56:15.054862 13119 raft_consensus.cc:2781] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:15.161643 13136 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 09:56:15.199618 13136 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:15.200030 13136 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:15.201138 13248 raft_consensus.cc:3032] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Stepping down as leader of term 8
I20240426 09:56:15.201516 13248 raft_consensus.cc:729] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
W20240426 09:56:15.138065 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 143986us
I20240426 09:56:15.202885 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:15.203612 10620 raft_consensus.cc:2379] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 10: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:56:15.204357 12983 consensus_queue.cc:260] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:15.018805 12950 consensus_queue.cc:1035] T 8574d769068f477e859574c959d5880e P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20240426 09:56:15.141894 9979 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
W20240426 09:56:15.211658 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 69853.6us
I20240426 09:56:15.211791 12985 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:15.212617 13196 raft_consensus.cc:3037] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:15.213866 9979 leader_election.cc:398] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8.
W20240426 09:56:15.133002 13264 log.cc:924] Time spent T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20240426 09:56:15.158385 10056 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:15.215359 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
W20240426 09:56:15.215585 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 57300.1us
W20240426 09:56:15.164230 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 49325.6us
I20240426 09:56:15.216557 13128 raft_consensus.cc:2781] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:56:15.216797 13227 raft_consensus.cc:2781] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:56:15.217170 13227 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:15.217305 13128 raft_consensus.cc:686] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [term 7 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:15.217558 13227 raft_consensus.cc:3037] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:15.218292 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:15.218953 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:15.220283 13237 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:15.220899 10084 raft_consensus.cc:1261] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20240426 09:56:15.220767 13237 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:15.219621 10089 raft_consensus.cc:2454] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:15.223251 13145 consensus_queue.cc:237] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:15.227766 10081 raft_consensus.cc:2396] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
W20240426 09:56:15.180027 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 146257us
I20240426 09:56:15.228499 9982 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:15.137156 13256 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 8
I20240426 09:56:15.161819 13229 raft_consensus.cc:729] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:56:15.214094 13160 consensus_queue.cc:260] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:15.191318 13046 consensus_meta.cc:287] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.768s user 0.005s sys 0.015s
I20240426 09:56:15.191494 13119 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:15.165616 13277 consensus_queue.cc:1035] T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20240426 09:56:15.195264 13129 raft_consensus.cc:686] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:15.196884 12991 tablet_bootstrap.cc:492] T 1b76245317c24e129e7f16cdb4d99007 P 1fee3da01ebc463396397057fdb81a68: Bootstrap starting.
I20240426 09:56:15.199167 13122 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:56:15.183233 13271 log.cc:924] Time spent T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20240426 09:56:15.219517 13128 consensus_queue.cc:237] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:15.245636 13119 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:15.258409 13122 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:15.231721 9979 leader_election.cc:401] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
W20240426 09:56:15.259935 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 46186.1us
I20240426 09:56:15.260428 13122 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:15.260684 9979 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:15.232537 9904 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:15.023149 13255 consensus_queue.cc:1046] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20240426 09:56:15.264348 13239 raft_consensus.cc:2726] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8.
I20240426 09:56:15.264954 13239 raft_consensus.cc:2726] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 09:56:15.165544 13207 consensus_queue.cc:1046] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.070s
I20240426 09:56:15.222661 13237 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:15.256394 13136 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:15.263784 9979 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:15.267429 13129 consensus_queue.cc:237] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:15.269480 13233 raft_consensus.cc:2781] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:15.269848 13233 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:15.270159 13233 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:15.271147 13229 consensus_queue.cc:260] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:15.247334 10089 raft_consensus.cc:1261] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20240426 09:56:15.272583 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:15.273180 10014 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:15.272758 13194 consensus_queue.cc:237] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:15.275007 9979 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:15.275508 13136 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 11 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:15.187985 13259 consensus_queue.cc:1046] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20240426 09:56:15.279523 10014 raft_consensus.cc:2454] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 11.
I20240426 09:56:15.278491 10089 pending_rounds.cc:76] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee: Aborting all ops after (but not including) 5
I20240426 09:56:15.280061 10089 pending_rounds.cc:97] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20240426 09:56:15.280336 10089 raft_consensus.cc:2866] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 09:56:15.277968 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 49553.1us
I20240426 09:56:15.281808 9982 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:15.225883 13148 consensus_queue.cc:1035] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20240426 09:56:15.289228 10585 leader_election.cc:302] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:56:15.288645 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:15.251252 13257 raft_consensus.cc:2741] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
W20240426 09:56:15.278999 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 267866us
I20240426 09:56:15.231534 12950 consensus_queue.cc:1046] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.068s
I20240426 09:56:15.295048 10086 raft_consensus.cc:1261] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20240426 09:56:15.295419 10086 pending_rounds.cc:76] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee: Aborting all ops after (but not including) 5
I20240426 09:56:15.295616 10086 pending_rounds.cc:97] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee: Aborting uncommitted NO_OP operation due to leader change: 9.6
W20240426 09:56:15.295679 10586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 109493us
I20240426 09:56:15.237380 13226 consensus_queue.cc:1035] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20240426 09:56:15.291615 10015 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 7.
I20240426 09:56:15.286871 13202 raft_consensus.cc:2781] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:15.304320 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:15.297580 10086 raft_consensus.cc:2866] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:56:15.305063 9933 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 8.
I20240426 09:56:15.298199 9905 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:15.306571 9905 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:15.272511 13229 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:15.281008 13239 raft_consensus.cc:2781] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:15.202283 13248 consensus_queue.cc:260] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:15.301491 10585 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:56:15.292697 10055 leader_election.cc:302] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:15.289357 13236 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:15.312012 13239 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:15.314900 13239 raft_consensus.cc:3037] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:15.298568 13207 consensus_queue.cc:1225] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer f12521b6d51242629cf4297864df00b9 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240426 09:56:15.298432 13136 raft_consensus.cc:2781] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 09:56:15.288066 12917 consensus_queue.cc:1035] T 69700f6b120b4f4ebc0b43b14166f4c9 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.021s
I20240426 09:56:15.303686 13244 raft_consensus.cc:3032] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Stepping down as leader of term 9
I20240426 09:56:15.329488 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:15.309674 13255 raft_consensus.cc:3032] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Stepping down as leader of term 8
I20240426 09:56:15.304718 13202 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:15.309448 13123 raft_consensus.cc:2726] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 09:56:15.349349 10055 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:15.309567 12976 raft_consensus.cc:2726] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 09:56:15.293222 13152 consensus_queue.cc:1035] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20240426 09:56:15.317494 13240 raft_consensus.cc:2726] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:56:15.406313 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:15.434322 13242 raft_consensus.cc:2726] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:15.434922 12917 raft_consensus.cc:2726] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 09:56:15.435581 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 86322.7us
W20240426 09:56:15.435976 13282 log.cc:924] Time spent T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20240426 09:56:15.436303 13160 raft_consensus.cc:3037] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
W20240426 09:56:15.439582 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 88554us
I20240426 09:56:15.434342 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:15.434576 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:15.441874 13286 consensus_queue.cc:1046] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 09:56:15.442873 13286 raft_consensus.cc:3032] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:56:15.443187 13286 raft_consensus.cc:729] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:56:15.331153 13230 raft_consensus.cc:2781] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:15.329339 13193 raft_consensus.cc:3032] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 4 LEADER]: Stepping down as leader of term 4
W20240426 09:56:15.446805 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 165051us
I20240426 09:56:15.328305 13244 raft_consensus.cc:729] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:15.451174 10612 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 09:56:15.332156 13136 raft_consensus.cc:686] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 11 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:56:15.349506 13202 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
W20240426 09:56:15.448544 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 167006us
I20240426 09:56:15.447939 9982 leader_election.cc:398] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8.
I20240426 09:56:15.331914 13255 raft_consensus.cc:729] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:15.440272 13256 raft_consensus.cc:2726] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821) with higher term. Message: Invalid argument: T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8.
I20240426 09:56:15.458784 9982 leader_election.cc:302] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:15.409757 13289 raft_consensus.cc:2726] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:15.451119 13193 raft_consensus.cc:729] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:56:15.484582 9982 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:15.447196 13230 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 09:56:15.456058 13270 log.cc:924] Time spent T 2e197425a8c74512b4dac9d31a3ce0a5 P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.286s user 0.000s sys 0.001s
I20240426 09:56:15.488866 10090 raft_consensus.cc:1261] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20240426 09:56:15.563424 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 80005us
I20240426 09:56:15.455582 13149 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:15.480177 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:15.458182 13242 consensus_queue.cc:1046] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.104s
W20240426 09:56:15.577718 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 136847us
I20240426 09:56:15.450893 13286 consensus_queue.cc:260] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:15.490453 13230 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:15.578722 9906 leader_election.cc:398] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 election: Vote denied by peer 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411) with higher term. Message: Invalid argument: T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 for earlier term 6. Current term is 7.
I20240426 09:56:15.489882 13256 raft_consensus.cc:2726] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 09:56:15.585189 10055 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
W20240426 09:56:15.586385 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 101870us
I20240426 09:56:15.482637 13123 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.095s
W20240426 09:56:15.568837 13268 log.cc:924] Time spent T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20240426 09:56:15.594062 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:15.594532 10009 raft_consensus.cc:1226] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 10. Current term is 11. Ops: []
I20240426 09:56:15.595618 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:15.603029 10012 raft_consensus.cc:1261] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 09:56:15.604586 10008 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:15.604996 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:15.604933 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:15.606050 9906 leader_election.cc:302] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:15.578187 13196 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:15.634564 10014 raft_consensus.cc:1261] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 09:56:15.634454 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:15.635435 10016 raft_consensus.cc:1261] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20240426 09:56:15.635983 10012 raft_consensus.cc:1261] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 09:56:15.742116 10012 raft_consensus.cc:1261] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:56:15.743359 10012 raft_consensus.cc:1261] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 09:56:15.638703 10010 raft_consensus.cc:1261] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 09:56:15.452471 13290 consensus_queue.cc:1046] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.146s
I20240426 09:56:15.660818 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:15.751180 10011 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:56:15.592451 12917 raft_consensus.cc:2726] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:15.562637 13227 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:15.603137 13207 raft_consensus.cc:2726] T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 09:56:15.774354 10007 raft_consensus.cc:1261] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:56:15.697204 10008 raft_consensus.cc:2454] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 9.
I20240426 09:56:15.587110 13240 consensus_queue.cc:1046] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
I20240426 09:56:15.588816 13149 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:15.642544 13152 raft_consensus.cc:2781] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 09:56:15.668893 13049 consensus_queue.cc:1046] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.092s
I20240426 09:56:15.662853 13196 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:15.740362 10009 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
W20240426 09:56:15.739593 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 72715us
I20240426 09:56:15.702441 13226 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:15.775084 10012 raft_consensus.cc:1261] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20240426 09:56:15.782161 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 120300us
W20240426 09:56:15.710932 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 120411us
I20240426 09:56:15.803403 9982 leader_election.cc:398] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 9. Current term is 10.
I20240426 09:56:15.803682 9982 leader_election.cc:401] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20240426 09:56:15.584156 13255 consensus_queue.cc:260] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:15.740830 10014 raft_consensus.cc:1261] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20240426 09:56:15.568459 13244 consensus_queue.cc:260] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:15.796392 10055 leader_election.cc:302] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:15.800122 12983 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:15.801676 10015 raft_consensus.cc:1226] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 4. Current term is 5. Ops: []
I20240426 09:56:15.669672 13233 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:15.705838 13236 consensus_queue.cc:1035] T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
I20240426 09:56:15.805560 9906 leader_election.cc:302] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:15.811619 10055 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:15.662559 13136 consensus_queue.cc:237] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:15.783073 13207 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:15.821978 10055 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:15.829494 10055 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
W20240426 09:56:15.818017 13291 log.cc:924] Time spent T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.362s user 0.001s sys 0.000s
I20240426 09:56:15.635773 13193 consensus_queue.cc:260] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:15.809365 13310 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:15.836180 10055 leader_election.cc:302] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
W20240426 09:56:15.838156 9979 leader_election.cc:334] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 2.093s (SENT)
I20240426 09:56:15.807435 13152 raft_consensus.cc:686] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [term 10 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
W20240426 09:56:15.837874 10585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 252311us
I20240426 09:56:15.820828 13285 raft_consensus.cc:2726] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:56:15.852367 9979 leader_election.cc:302] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:15.821009 13292 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:15.886308 9941 raft_consensus.cc:1261] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20240426 09:56:15.809533 13226 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:15.897073 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 58937.9us
I20240426 09:56:15.838403 13128 raft_consensus.cc:2726] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240426 09:56:15.859187 13300 log.cc:924] Time spent T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.291s user 0.000s sys 0.001s
I20240426 09:56:15.854585 13311 raft_consensus.cc:2726] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 09:56:15.888185 13129 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:15.837740 13207 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:15.935678 13152 consensus_queue.cc:237] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:15.909582 13291 log.cc:924] Time spent T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
W20240426 09:56:15.915428 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 112172us
I20240426 09:56:15.941125 9982 leader_election.cc:302] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:15.942530 9982 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
W20240426 09:56:15.912119 13303 log.cc:924] Time spent T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.131s user 0.001s sys 0.000s
I20240426 09:56:15.918179 13317 raft_consensus.cc:2726] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 09:56:15.925659 13307 log.cc:924] Time spent T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
W20240426 09:56:15.926347 13301 log.cc:924] Time spent T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.145s user 0.000s sys 0.000s
I20240426 09:56:15.927085 13149 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:15.927340 13196 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:15.928234 13299 log.cc:924] Time spent T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.260s user 0.000s sys 0.000s
I20240426 09:56:15.928393 13233 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:15.947901 13233 raft_consensus.cc:2726] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 09:56:15.928998 13290 raft_consensus.cc:2726] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:56:15.898134 13292 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:15.948122 13311 consensus_queue.cc:1035] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 09:56:15.855420 13310 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:15.949605 13311 consensus_queue.cc:1035] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:56:15.951174 13310 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:15.952232 13310 raft_consensus.cc:2726] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 09:56:15.929950 13312 raft_consensus.cc:3032] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:56:15.952976 13312 raft_consensus.cc:729] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:15.953862 13312 consensus_queue.cc:260] T 6cfada00f42a411f813bfc35d6c9ec9d P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:15.930893 13129 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:15.960011 10081 raft_consensus.cc:1261] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240426 09:56:15.960480 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:15.961182 10086 raft_consensus.cc:2379] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 5: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:15.930630 13240 raft_consensus.cc:2726] T 5e11de7e65e544ffac5f7683e9614536 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113) with higher term. Message: Invalid argument: T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 9. Current term is 10.
I20240426 09:56:15.961537 13311 consensus_queue.cc:1035] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 09:56:15.961182 10081 raft_consensus.cc:1261] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 09:56:15.930143 13146 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:15.965735 13146 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:15.967727 13146 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:15.932461 13257 raft_consensus.cc:2726] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240426 09:56:15.933887 13239 consensus_meta.cc:287] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.619s user 0.008s sys 0.000s
I20240426 09:56:15.969199 13239 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:15.930089 13227 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:15.934057 13298 log.cc:924] Time spent T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.273s user 0.001s sys 0.000s
I20240426 09:56:15.974505 9982 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:56:15.977581 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 142816us
W20240426 09:56:15.983438 10088 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.017s sys 0.014s
W20240426 09:56:15.987959 10088 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s user 0.017s sys 0.014s
I20240426 09:56:15.919849 13236 consensus_queue.cc:1035] T ac0131f6d1d54716ab3324104a18c9c3 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.065s
I20240426 09:56:15.949646 13292 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:56:15.984418 13119 consensus_meta.cc:287] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.724s user 0.007s sys 0.003s
I20240426 09:56:16.018462 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:16.020974 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:16.020179 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:16.024379 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:16.020694 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:16.025388 9938 raft_consensus.cc:2379] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 4: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:16.020867 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:16.026943 9937 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:16.027383 9906 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:16.028506 9906 leader_election.cc:302] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:16.028743 9933 raft_consensus.cc:2379] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 7: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:16.033782 9937 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:56:16.014236 13227 raft_consensus.cc:2781] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:16.038981 13227 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:16.039376 13227 raft_consensus.cc:3037] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:16.039530 9939 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 13 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20240426 09:56:16.046875 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:16.047164 13226 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:16.047693 13226 consensus_queue.cc:1035] T cfa61a21874f49688b31ddd60646a658 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:56:16.049552 10083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 1928) took 1126 ms. Trace:
W20240426 09:56:16.050395 9979 leader_election.cc:334] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 2.139s (SENT)
I20240426 09:56:16.051244 9979 leader_election.cc:302] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:16.051999 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:16.049914 10083 rpcz_store.cc:276] 0426 09:56:14.922850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:14.923171 (+ 321us) service_pool.cc:225] Handling call
0426 09:56:14.923502 (+ 331us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:14.926032 (+ 2530us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 09:56:15.039303 (+113271us) log.cc:842] Serialized 14 byte log entry
0426 09:56:15.251466 (+212163us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:15.251839 (+ 373us) log.cc:842] Serialized 26 byte log entry
0426 09:56:15.251949 (+ 110us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:56:15.251980 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:15.252045 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:15.929919 (+677874us) spinlock_profiling.cc:243] Waited 5.4 ms on lock 0x7b48000b0c58. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca7aee2e1 00007f9ca7ae53d7 00007f9ca7ae526a 00007f9ca7bd943f 00007f9ca7bd9107 00007f9ca7bed60c 00007f9ca7bed5b8 00007f9ca7bed538 00007f9ca7bed4f3 00007f9ca7bec79c 00007f9ca6b6f8d6
0426 09:56:16.049293 (+119374us) raft_consensus.cc:1671] finished
0426 09:56:16.049391 (+ 98us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:16.049500 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16387,"prepare.queue_time_us":277041,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":34,"raft.queue_time_us":14263,"raft.run_cpu_time_us":7110,"raft.run_wall_time_us":672283,"spinlock_wait_cycles":13037184,"thread_start_us":204238,"threads_started":2,"wal-append.queue_time_us":284621}
I20240426 09:56:16.057826 9940 raft_consensus.cc:1226] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [term 10 LEADER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 9. Current term is 10. Ops: []
W20240426 09:56:16.061901 9979 leader_election.cc:334] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 2.158s (SENT)
I20240426 09:56:16.062361 9979 leader_election.cc:302] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:16.015568 13207 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:16.038859 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:16.009832 13311 consensus_queue.cc:1035] T 2f96620cc62e4bf1beefad6b59ca3968 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
W20240426 09:56:16.101629 9979 leader_election.cc:334] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 2.210s (SENT)
I20240426 09:56:16.018189 13257 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:16.035228 13285 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:16.118243 13285 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:16.120357 13285 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:16.049973 13129 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:16.107017 10009 raft_consensus.cc:1226] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 4. Current term is 5. Ops: []
I20240426 09:56:16.107754 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:16.012463 13117 consensus_queue.cc:1035] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.052s
I20240426 09:56:16.060490 13226 raft_consensus.cc:2781] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:16.111122 13318 raft_consensus.cc:2726] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 09:56:16.117663 10616 raft_consensus.cc:2454] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:56:16.123474 10016 raft_consensus.cc:2454] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 3.
I20240426 09:56:16.065833 13239 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:16.064868 13240 raft_consensus.cc:2726] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 09:56:16.041769 13149 consensus_queue.cc:1035] T 17e4aa133eb141699b9a5e2f0b4767d6 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20240426 09:56:16.098484 13245 raft_consensus.cc:2726] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20240426 09:56:16.131623 9979 leader_election.cc:334] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 2.270s (SENT)
I20240426 09:56:16.028002 13119 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:16.131179 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 101541us
W20240426 09:56:16.155722 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 319178us
I20240426 09:56:16.153964 10585 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:56:16.158597 10055 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:16.130000 13226 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:16.102099 13128 consensus_queue.cc:1035] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.053s
I20240426 09:56:16.169438 10055 leader_election.cc:302] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:16.175082 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:56:16.170125 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 141693us
I20240426 09:56:16.156294 9980 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:56:16.054085 13319 consensus_queue.cc:1035] T d9f79bd8994f4b229ca8da8638bb6d5a P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.117s
I20240426 09:56:16.125236 13257 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:16.053175 13324 consensus_queue.cc:1035] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.117s
I20240426 09:56:16.176211 9906 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:16.172152 10013 raft_consensus.cc:1226] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 6. Current term is 7. Ops: []
I20240426 09:56:16.175710 10083 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:56:16.168785 13289 raft_consensus.cc:2726] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 09:56:16.212126 13119 leader_election.cc:288] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Requested vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:16.174315 13226 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Advancing to term 7
W20240426 09:56:16.182940 13229 consensus_meta.cc:287] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68: Time spent flushing consensus metadata: real 0.866s user 0.010s sys 0.000s
W20240426 09:56:16.196949 13160 consensus_meta.cc:287] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.760s user 0.005s sys 0.005s
I20240426 09:56:16.219802 10084 raft_consensus.cc:1261] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:56:16.220968 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:16.221714 13257 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:16.222143 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:16.175668 13327 consensus_queue.cc:1035] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20240426 09:56:16.225015 10016 raft_consensus.cc:1226] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:16.225484 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007f1451130c4dfcbd86145d56a6f67e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:16.226047 10090 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:16.196591 13316 raft_consensus.cc:2781] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:56:16.226694 13316 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:16.227068 13316 raft_consensus.cc:3037] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:56:16.228169 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 52041.3us
I20240426 09:56:16.154151 13236 raft_consensus.cc:2781] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 09:56:16.204102 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:16.231272 10083 raft_consensus.cc:3037] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:16.205085 13309 raft_consensus.cc:2781] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:56:16.205524 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:16.155911 13323 consensus_queue.cc:1046] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20240426 09:56:16.238137 10012 raft_consensus.cc:1226] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 8. Current term is 9. Ops: []
I20240426 09:56:16.238291 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e117a0ad5cd34acea96de04537a80733" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:16.241493 10014 raft_consensus.cc:2379] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 8: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:56:16.241739 10008 raft_consensus.cc:1261] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20240426 09:56:16.236079 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:16.308295 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:16.314829 10008 raft_consensus.cc:2454] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 10.
I20240426 09:56:16.182832 13285 raft_consensus.cc:2726] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 09:56:16.323911 13275 consensus_queue.cc:1046] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20240426 09:56:16.324824 10011 raft_consensus.cc:1261] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:56:16.333334 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:16.334146 10089 raft_consensus.cc:2379] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 7: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
W20240426 09:56:16.333197 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 103483us
I20240426 09:56:16.336620 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
W20240426 09:56:16.341830 10584 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.010s sys 0.018s
W20240426 09:56:16.351135 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 180962us
W20240426 09:56:16.384341 10584 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.547s user 0.013s sys 0.020s
W20240426 09:56:16.385241 9979 leader_election.cc:334] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 2.496s (SENT)
I20240426 09:56:16.333173 10016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:16.240774 13309 raft_consensus.cc:482] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:16.385469 10083 raft_consensus.cc:2454] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:56:16.387727 10016 raft_consensus.cc:2379] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 7: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:56:16.327344 13283 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:16.386499 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a3a93bd7c449cf86ef18473507bf75" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:56:16.392354 9904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 223177us
I20240426 09:56:16.334825 13145 raft_consensus.cc:2781] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:16.396471 9979 leader_election.cc:302] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:16.398204 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:16.399634 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:16.401471 9979 leader_election.cc:302] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:16.402590 13239 raft_consensus.cc:2726] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 09:56:16.354892 13236 raft_consensus.cc:686] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [term 9 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:16.403909 9982 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:16.405269 10083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:16.383184 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:16.355048 13324 consensus_queue.cc:1035] T 5e11de7e65e544ffac5f7683e9614536 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20240426 09:56:16.397532 13145 raft_consensus.cc:482] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:16.476090 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:16.393884 13160 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:16.470010 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:16.479095 10082 raft_consensus.cc:2362] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 6. Re-sending same reply.
I20240426 09:56:16.477007 10089 raft_consensus.cc:2454] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 9.
W20240426 09:56:16.480906 13334 log.cc:924] Time spent T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20240426 09:56:16.403021 13328 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:16.389612 13309 raft_consensus.cc:3037] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Advancing to term 5
W20240426 09:56:16.489156 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 138552us
I20240426 09:56:16.489709 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:16.494882 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:16.495218 10012 raft_consensus.cc:3032] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 LEADER]: Stepping down as leader of term 9
I20240426 09:56:16.495759 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:16.496197 10012 raft_consensus.cc:729] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:16.498142 10620 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 13 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20240426 09:56:16.478381 13145 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:16.503080 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 14 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:16.503732 10082 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 13.
I20240426 09:56:16.504343 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:16.504962 10081 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 9.
I20240426 09:56:16.475584 13318 raft_consensus.cc:2781] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:56:16.496918 10012 consensus_queue.cc:260] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:16.510314 10012 raft_consensus.cc:3037] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:16.510903 9979 leader_election.cc:302] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:16.511600 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:16.512425 10089 raft_consensus.cc:2396] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240426 09:56:16.514110 10583 leader_election.cc:302] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:16.519002 10055 leader_election.cc:302] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:16.543774 10583 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:16.544943 10583 leader_election.cc:302] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:16.395638 13283 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:16.545997 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:16.550647 10612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:16.489914 13160 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:16.551184 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:16.551396 10055 leader_election.cc:302] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:16.552876 10055 leader_election.cc:302] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
W20240426 09:56:16.553488 13340 log.cc:924] Time spent T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20240426 09:56:16.554735 13325 raft_consensus.cc:2726] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:16.555507 13325 consensus_queue.cc:1035] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 09:56:16.556063 13128 raft_consensus.cc:2781] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 09:56:16.556490 13128 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:16.556855 13128 raft_consensus.cc:3037] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:16.557277 13325 consensus_queue.cc:1035] T 72a6c52e0c924ba3b3fca5f9586814a7 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 09:56:16.558883 13336 raft_consensus.cc:2726] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 09:56:16.559247 13229 raft_consensus.cc:2781] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 09:56:16.559695 13229 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
W20240426 09:56:16.546073 13230 consensus_meta.cc:287] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68: Time spent flushing consensus metadata: real 0.920s user 0.010s sys 0.000s
I20240426 09:56:16.560556 13230 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:16.512955 13202 consensus_meta.cc:287] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 1.033s user 0.009s sys 0.001s
I20240426 09:56:16.563041 13202 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:16.564903 13202 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:16.565616 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:16.566231 9935 raft_consensus.cc:2454] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 4.
I20240426 09:56:16.481861 13119 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.067s
I20240426 09:56:16.558890 13321 raft_consensus.cc:2726] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 09:56:16.568208 10619 raft_consensus.cc:2379] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 9: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:56:16.568588 13321 raft_consensus.cc:2726] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 09:56:16.569727 10613 raft_consensus.cc:1226] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [term 11 LEADER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 10. Current term is 11. Ops: []
I20240426 09:56:16.570055 13339 consensus_queue.cc:1035] T 50a3a93bd7c449cf86ef18473507bf75 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20240426 09:56:16.571069 13283 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:16.505311 13323 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:56:16.544615 12983 consensus_meta.cc:287] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.722s user 0.009s sys 0.000s
I20240426 09:56:16.579246 10012 raft_consensus.cc:1261] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20240426 09:56:16.631145 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 86282.5us
I20240426 09:56:16.631752 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:16.632436 10012 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 9.
I20240426 09:56:16.632545 10583 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:16.572346 10082 raft_consensus.cc:2362] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 10. Re-sending same reply.
I20240426 09:56:16.591101 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:16.560070 13229 raft_consensus.cc:3037] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:16.562582 13230 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:16.651494 10015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1956) took 1058 ms. Trace:
I20240426 09:56:16.655009 13345 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
W20240426 09:56:16.655675 13333 log.cc:924] Time spent T 60bebc132e4141bdbab199326986516e P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.265s user 0.000s sys 0.000s
I20240426 09:56:16.554613 13346 raft_consensus.cc:2781] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 09:56:16.669065 10015 rpcz_store.cc:276] 0426 09:56:15.593200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:15.593312 (+ 112us) service_pool.cc:225] Handling call
0426 09:56:15.633415 (+ 40103us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:15.818931 (+185516us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:16.651266 (+832335us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b540025e754. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:16.651440 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":372223232}
W20240426 09:56:16.629815 13227 consensus_meta.cc:287] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.583s user 0.005s sys 0.003s
I20240426 09:56:16.677042 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e197425a8c74512b4dac9d31a3ce0a5" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:16.548610 13117 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:16.710711 10081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:16.703823 10089 raft_consensus.cc:2354] T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 5. Current term is 6.
I20240426 09:56:16.656538 13288 raft_consensus.cc:2781] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 09:56:16.738946 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:16.506644 13328 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:16.675815 12983 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:16.742692 10613 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 10.
I20240426 09:56:16.680859 13283 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:16.698917 13245 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:16.742646 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dfdb63e28a460486dfe163f03bf820" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:16.787396 10089 raft_consensus.cc:3037] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
W20240426 09:56:16.746726 10090 consensus_meta.cc:287] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.520s user 0.000s sys 0.007s
I20240426 09:56:16.787967 10090 raft_consensus.cc:2454] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:56:16.715574 13336 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:16.790474 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:16.638072 13240 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.102s
I20240426 09:56:16.808962 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:16.737264 13346 raft_consensus.cc:686] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [term 6 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
W20240426 09:56:16.780413 9980 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 275241us
W20240426 09:56:16.784909 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 274054us
I20240426 09:56:16.739727 13288 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:16.647593 13323 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:16.639788 13236 consensus_queue.cc:237] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:16.787657 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:16.823009 13145 raft_consensus.cc:504] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:16.823920 13297 raft_consensus.cc:3032] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 09:56:16.824438 13297 raft_consensus.cc:729] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:56:16.825520 9935 raft_consensus.cc:2379] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1fee3da01ebc463396397057fdb81a68 in current term 9: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:16.825213 13297 consensus_queue.cc:260] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:16.826231 13297 raft_consensus.cc:3037] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:16.829418 10084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 1942) took 1602 ms. Trace:
I20240426 09:56:16.829805 10084 rpcz_store.cc:276] 0426 09:56:15.227181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:15.227326 (+ 145us) service_pool.cc:225] Handling call
0426 09:56:15.254377 (+ 27051us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:16.220161 (+965784us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:16.829212 (+609051us) spinlock_profiling.cc:243] Waited 965 ms on lock 0x7b540024f754. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:16.829366 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2219478400}
W20240426 09:56:16.834038 9979 leader_election.cc:334] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 2.553s (SENT)
I20240426 09:56:16.715912 13227 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:16.835855 13257 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:16.836314 13353 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:16.782146 13319 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:16.836828 13353 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:16.837296 13319 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:16.836328 13257 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:16.839320 13319 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:16.792105 13283 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:16.840922 13309 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:16.841485 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:16.786253 12983 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:16.811123 13226 consensus_meta.cc:287] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9: Time spent flushing consensus metadata: real 0.598s user 0.008s sys 0.000s
I20240426 09:56:16.837934 9979 leader_election.cc:302] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:16.843274 13257 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:16.843144 13226 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:16.843695 12983 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:16.844457 10619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51158 (request call id 513) took 1438 ms. Trace:
I20240426 09:56:16.824951 13145 leader_election.cc:288] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:16.845371 13240 raft_consensus.cc:2726] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 09:56:16.836715 13227 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:16.736549 13117 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:16.749377 13285 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:16.750097 13314 raft_consensus.cc:2726] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 09:56:16.814869 9906 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:16.784440 13344 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:56:16.785985 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 317033us
I20240426 09:56:16.873319 9982 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:16.716145 13345 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:16.808152 13245 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:16.812824 13336 raft_consensus.cc:504] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:16.816279 13288 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 13 FOLLOWER]: Advancing to term 14
I20240426 09:56:16.820727 12969 consensus_queue.cc:1035] T 9da46d724daf4bbf9c49a393e3579156 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240426 09:56:16.821370 13316 consensus_meta.cc:287] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.594s user 0.008s sys 0.000s
I20240426 09:56:16.841692 13283 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:16.807803 9940 raft_consensus.cc:3032] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 09:56:16.907971 9940 raft_consensus.cc:729] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:16.846984 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:16.844832 10619 rpcz_store.cc:276] 0426 09:56:15.405841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:15.406006 (+ 165us) service_pool.cc:225] Handling call
0426 09:56:16.844227 (+1438221us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54005528d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:16.844404 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2611208576}
I20240426 09:56:16.847272 9979 leader_election.cc:302] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:56:16.865614 13353 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:16.892238 13152 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 09:56:16.779820 13311 raft_consensus.cc:482] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:16.790891 13347 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:16.807935 12976 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:16.878129 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:16.878360 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:16.879968 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:16.917977 9938 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 10.
I20240426 09:56:16.817858 13203 raft_consensus.cc:482] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:16.896248 10612 raft_consensus.cc:2379] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 5: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:56:16.897958 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:16.922103 10013 raft_consensus.cc:3037] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 9
I20240426 09:56:16.895936 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:16.899194 10054 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:16.899574 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:16.923997 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:16.905500 9982 leader_election.cc:302] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:16.906404 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:16.909830 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:16.907950 10081 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
W20240426 09:56:16.911443 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 64240.1us
I20240426 09:56:16.925644 10090 raft_consensus.cc:2379] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 7: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:16.926034 9938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:16.927137 9979 leader_election.cc:302] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
W20240426 09:56:16.864674 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 49898.1us
I20240426 09:56:16.906143 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:16.912041 9940 consensus_queue.cc:260] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:16.900741 13339 raft_consensus.cc:482] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:16.925431 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:16.930647 9935 raft_consensus.cc:2379] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 8: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:16.929611 9937 raft_consensus.cc:2379] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 7: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:16.922438 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:16.933715 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:16.912958 13318 raft_consensus.cc:686] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:16.934875 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:16.935315 10089 raft_consensus.cc:2454] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:56:16.936311 9979 leader_election.cc:302] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68
I20240426 09:56:16.872366 13128 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:16.872095 13285 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:16.939525 9937 raft_consensus.cc:2379] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 7: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:16.930733 10619 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:56:16.914577 13152 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:16.999434 9936 raft_consensus.cc:1203] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Deduplicated request from leader. Original: 5.3->[6.4-6.4] Dedup: 6.4->[]
W20240426 09:56:16.975911 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 58662us
I20240426 09:56:17.000389 9936 raft_consensus.cc:1226] T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240426 09:56:16.942314 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:16.912811 13352 raft_consensus.cc:2726] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 09:56:16.973850 10055 leader_election.cc:302] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:17.004401 10054 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:17.003975 13257 raft_consensus.cc:2781] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:56:16.927839 13160 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:17.008496 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 78105.4us
I20240426 09:56:17.009789 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:16.930233 13123 consensus_queue.cc:1035] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
I20240426 09:56:17.011045 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:17.006253 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:17.037977 10089 raft_consensus.cc:2454] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:56:17.038933 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46890d4f3474e88835f7b3e08a30e1a" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:17.039527 10089 raft_consensus.cc:3037] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:17.007237 10090 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8"
dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
mode: GRACEFUL
new_leader_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
from {username='slave'} at 127.0.0.1:50408
I20240426 09:56:17.043072 10090 raft_consensus.cc:595] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Received request to transfer leadership to TS 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:17.043434 10090 raft_consensus.cc:603] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 09:56:17.007680 9935 raft_consensus.cc:1226] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 6. Current term is 7. Ops: []
I20240426 09:56:17.011613 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:16.928682 13353 raft_consensus.cc:2726] T 50a3a93bd7c449cf86ef18473507bf75 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 09:56:17.023636 9936 raft_consensus.cc:2379] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 4: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:16.931594 13325 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 09:56:16.908094 13316 raft_consensus.cc:504] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:16.910192 13276 consensus_queue.cc:1046] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.102s
I20240426 09:56:17.057622 9982 leader_election.cc:302] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:16.928202 13350 raft_consensus.cc:2781] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:56:17.051331 10089 raft_consensus.cc:2454] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 8.
I20240426 09:56:17.004282 13152 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:17.009660 13319 consensus_queue.cc:1046] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20240426 09:56:16.923331 12976 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:17.076942 10056 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9
I20240426 09:56:17.053603 13160 raft_consensus.cc:2726] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 09:56:16.897260 13344 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.074959 10014 raft_consensus.cc:2454] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:56:17.073141 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:17.068521 9904 leader_election.cc:302] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:16.929939 13203 raft_consensus.cc:504] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.069296 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:17.069872 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308d6ed4b68346be85caf2871916eab9" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:17.077328 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:17.086467 9936 raft_consensus.cc:3037] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:17.053319 13257 raft_consensus.cc:686] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:16.921869 13311 raft_consensus.cc:504] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:17.061340 13325 raft_consensus.cc:482] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:17.085719 10085 raft_consensus.cc:3032] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:56:16.941699 13339 raft_consensus.cc:504] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.075702 13240 raft_consensus.cc:2726] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20240426 09:56:17.087509 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 83195.6us
I20240426 09:56:17.090929 13285 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:17.091264 10056 leader_election.cc:302] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:17.072551 12983 raft_consensus.cc:2781] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 09:56:17.063745 13359 raft_consensus.cc:2726] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 09:56:17.095582 10090 raft_consensus.cc:1226] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 9. Current term is 10. Ops: []
I20240426 09:56:17.093044 10085 raft_consensus.cc:729] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:17.077817 13350 raft_consensus.cc:686] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Becoming Leader. State: Replica: 826f03b2fb9c4b9483623b041a1bf80e, State: Running, Role: LEADER
I20240426 09:56:17.082630 13226 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:17.119184 10084 raft_consensus.cc:1261] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 13 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20240426 09:56:17.087777 13328 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:17.125842 9939 raft_consensus.cc:1261] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Refusing update from remote peer d3a480e831d649139ff016fb68aee3ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:56:16.923640 13347 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:17.097015 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1960) took 1502 ms. Trace:
I20240426 09:56:17.166059 10007 rpcz_store.cc:276] 0426 09:56:15.594258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:15.594885 (+ 627us) service_pool.cc:225] Handling call
0426 09:56:15.595212 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:15.774718 (+179506us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:17.096816 (+1322098us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b54000587d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:17.096962 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":411244288}
I20240426 09:56:16.930711 13346 consensus_queue.cc:237] T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:17.148391 9941 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
W20240426 09:56:17.169587 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 98748.3us
I20240426 09:56:17.094368 13325 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:17.128053 10085 consensus_queue.cc:260] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.173763 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:17.173897 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:17.092037 13283 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:17.177404 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:17.123170 13117 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:17.187391 13283 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:17.191740 13283 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:17.192881 13283 raft_consensus.cc:2726] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 09:56:17.118183 13324 raft_consensus.cc:2781] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 09:56:17.122133 13309 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:56:17.201143 10583 leader_election.cc:334] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821): Timed out: RequestConsensusVote RPC to 127.8.141.131:34821 timed out after 1.975s (SENT)
W20240426 09:56:17.177724 13361 log.cc:924] Time spent T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20240426 09:56:17.210721 10583 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:17.126138 12983 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:17.125077 13240 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:17.211246 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:17.199539 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:17.197355 13226 raft_consensus.cc:482] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:17.177954 13245 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:17.174472 13366 raft_consensus.cc:2726] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 09:56:17.200009 13353 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:17.207407 13323 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:17.234947 10090 raft_consensus.cc:2454] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 5.
I20240426 09:56:17.092257 13318 consensus_queue.cc:237] T e117a0ad5cd34acea96de04537a80733 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.209465 13257 consensus_queue.cc:237] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:17.471462 10086 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 1954) took 1511 ms. Trace:
I20240426 09:56:17.258028 13350 consensus_queue.cc:237] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:17.476236 10086 rpcz_store.cc:276] 0426 09:56:15.959945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:15.961750 (+ 1805us) service_pool.cc:225] Handling call
0426 09:56:15.962006 (+ 256us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:15.966895 (+ 4889us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:15.966922 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:15.967267 (+ 345us) log.cc:842] Serialized 26 byte log entry
0426 09:56:16.063133 (+ 95866us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:56:16.063172 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:17.252475 (+1189303us) spinlock_profiling.cc:243] Waited 4.57 ms on lock 0x7b54002f1a54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:17.470915 (+218440us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:17.470974 (+ 59us) raft_consensus.cc:1671] finished
0426 09:56:17.470990 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:17.471248 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1092,"raft.run_cpu_time_us":63,"raft.run_wall_time_us":63,"spinlock_wait_cycles":10509696,"thread_start_us":95528,"threads_started":1,"wal-append.queue_time_us":208175,"wal-append.run_cpu_time_us":1548,"wal-append.run_wall_time_us":1025235}
I20240426 09:56:17.478514 13386 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:17.234180 12983 raft_consensus.cc:3037] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:17.418087 13311 leader_election.cc:288] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:17.479421 13386 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.301478 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:17.403813 13344 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:17.234710 13352 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:17.466181 13325 raft_consensus.cc:504] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:17.395659 13367 log.cc:924] Time spent T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.271s user 0.000s sys 0.001s
I20240426 09:56:17.475234 13390 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:17.245918 13128 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 11 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:17.234462 13336 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:17.237501 13240 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.341112 13148 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:17.495533 13148 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.496487 13240 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:17.496660 13294 log.cc:924] Time spent T 50a3a93bd7c449cf86ef18473507bf75 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 1.928s user 0.006s sys 0.008s
I20240426 09:56:17.497475 13148 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:17.497922 10087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 1935) took 2445 ms. Trace:
W20240426 09:56:17.339290 13370 log.cc:924] Time spent T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.133s user 0.000s sys 0.001s
I20240426 09:56:17.500660 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1984) took 1006 ms. Trace:
I20240426 09:56:17.498152 10012 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1981) took 1004 ms. Trace:
I20240426 09:56:17.418644 10013 raft_consensus.cc:2454] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 9.
I20240426 09:56:17.417421 9936 raft_consensus.cc:2454] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 5.
I20240426 09:56:17.416211 10090 raft_consensus.cc:1226] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20240426 09:56:17.447598 10016 raft_consensus.cc:1226] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 5. Current term is 6. Ops: []
I20240426 09:56:17.447093 10011 raft_consensus.cc:1261] T 28378f040d6642db8ba229c32bfed21a P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20240426 09:56:17.255218 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 58746us
W20240426 09:56:17.479131 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 387908us
I20240426 09:56:17.482357 13352 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:17.248512 13226 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:17.495016 13357 log.cc:924] Time spent T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.414s user 0.002s sys 0.000s
I20240426 09:56:17.498296 10087 rpcz_store.cc:276] 0426 09:56:15.052733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:15.153402 (+100669us) service_pool.cc:225] Handling call
0426 09:56:15.153830 (+ 428us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:15.159479 (+ 5649us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:15.159533 (+ 54us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:15.171672 (+ 12139us) log.cc:842] Serialized 26 byte log entry
0426 09:56:15.303487 (+131815us) raft_consensus.cc:1631] Marking committed up to 4
0426 09:56:15.303523 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:15.303583 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:17.496325 (+2192742us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b24001e0cc8. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fb76ce4 00007f9c9fb74dfb 00007f9c9fc02e09 00007f9c9fc02a95 00007f9c9fc020dd 00007f9ca28b12ee 00007f9ca28c8a8c 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1
0426 09:56:17.497641 (+ 1316us) raft_consensus.cc:1671] finished
0426 09:56:17.497661 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:17.497869 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4898,"raft.queue_time_us":37243,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":1584,"spinlock_wait_cycles":25628032,"thread_start_us":108999,"threads_started":1,"wal-append.queue_time_us":267827}
I20240426 09:56:17.502099 10012 rpcz_store.cc:276] 0426 09:56:16.493383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:16.633027 (+139644us) service_pool.cc:225] Handling call
0426 09:56:16.633333 (+ 306us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:16.649283 (+ 15950us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 09:56:16.913920 (+264637us) log.cc:842] Serialized 14 byte log entry
0426 09:56:17.001804 (+ 87884us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:17.013521 (+ 11717us) log.cc:842] Serialized 26 byte log entry
0426 09:56:17.013637 (+ 116us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:56:17.013664 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:17.013724 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:17.497967 (+484243us) raft_consensus.cc:1671] finished
0426 09:56:17.497984 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:17.498100 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21239,"prepare.queue_time_us":185761,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"raft.queue_time_us":21434,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":34434,"thread_start_us":276525,"threads_started":2,"wal-append.queue_time_us":158114}
I20240426 09:56:17.478600 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:17.508037 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:17.504345 10056 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee; no voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9
I20240426 09:56:17.338256 13355 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:17.501199 10008 rpcz_store.cc:276] 0426 09:56:16.494034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:16.973349 (+479315us) service_pool.cc:225] Handling call
0426 09:56:16.973735 (+ 386us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:16.999916 (+ 26181us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:16.999945 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:17.000363 (+ 418us) log.cc:842] Serialized 26 byte log entry
0426 09:56:17.070333 (+ 69970us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:56:17.070378 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:17.070446 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:17.500403 (+429957us) raft_consensus.cc:1671] finished
0426 09:56:17.500429 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:17.500608 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25807,"raft.queue_time_us":26198,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":5015,"spinlock_wait_cycles":1336192,"thread_start_us":56204,"threads_started":1,"wal-append.queue_time_us":167384}
W20240426 09:56:17.396039 13229 consensus_meta.cc:287] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68: Time spent flushing consensus metadata: real 0.686s user 0.007s sys 0.001s
I20240426 09:56:17.518601 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:17.520403 10008 raft_consensus.cc:2454] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 10.
I20240426 09:56:17.519793 13229 raft_consensus.cc:504] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:17.521209 10619 raft_consensus.cc:1226] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 9. Current term is 10. Ops: []
I20240426 09:56:17.483331 13390 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.506233 10090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 633) took 1086 ms. Trace:
I20240426 09:56:17.522976 10090 rpcz_store.cc:276] 0426 09:56:16.420143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.415680 (+995537us) service_pool.cc:225] Handling call
0426 09:56:17.416000 (+ 320us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:17.505924 (+ 89924us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:17.506178 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:17.523430 13390 leader_election.cc:288] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:17.524564 13339 leader_election.cc:288] T fa1ede5c9b614a0f8b7adb20a7cfbbb1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:17.519683 13226 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:17.527230 13286 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:17.528293 13229 leader_election.cc:288] T 472a3b5debc94412b63080f474ae4fed P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 10 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:17.528729 13289 raft_consensus.cc:3037] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:17.530539 13128 consensus_queue.cc:1046] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 6.4, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 09:56:17.531327 13128 consensus_queue.cc:1225] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Peer f12521b6d51242629cf4297864df00b9 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240426 09:56:17.531802 13128 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:17.520061 13355 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:17.532294 13128 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.534138 13128 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:17.531638 13242 consensus_queue.cc:1046] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20240426 09:56:17.535701 13242 raft_consensus.cc:3032] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:56:17.300592 13353 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:17.536058 13242 raft_consensus.cc:729] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:17.536918 13242 consensus_queue.cc:260] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.538046 13242 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:17.539031 10007 raft_consensus.cc:2454] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 2.
I20240426 09:56:17.539858 9940 raft_consensus.cc:3037] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:17.543334 10014 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34390 (request call id 1978) took 1050 ms. Trace:
I20240426 09:56:17.544075 13353 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:17.531590 10016 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20240426 09:56:17.546844 9940 raft_consensus.cc:1261] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Refusing update from remote peer 1fee3da01ebc463396397057fdb81a68: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20240426 09:56:17.505627 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1986) took 1011 ms. Trace:
I20240426 09:56:17.517019 13380 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:17.518008 13352 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:17.399621 13288 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:17.249786 13230 raft_consensus.cc:2726] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 09:56:17.518285 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 2464) took 2223 ms. Trace:
I20240426 09:56:17.395426 13203 leader_election.cc:288] T 72a6c52e0c924ba3b3fca5f9586814a7 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:17.538168 13248 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:17.362252 13316 leader_election.cc:288] T 951e4c48fcb642e58ab79652d3f86261 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:17.487884 13255 raft_consensus.cc:3037] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:17.318832 13117 consensus_queue.cc:1035] T 253cc496ee9644ffba6201074808dbf0 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.253s
W20240426 09:56:17.557509 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 461550us
I20240426 09:56:17.519013 9937 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 7.
W20240426 09:56:17.519362 13392 log.cc:924] Time spent T 5fc982aface54033a68641d8d809fd3b P 1fee3da01ebc463396397057fdb81a68: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20240426 09:56:17.537246 10082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 627) took 1146 ms. Trace:
I20240426 09:56:17.543689 10014 rpcz_store.cc:276] 0426 09:56:16.492623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:16.495518 (+ 2895us) service_pool.cc:225] Handling call
0426 09:56:17.543187 (+1047669us) spinlock_profiling.cc:243] Waited 578 ms on lock 0x7b540004f1d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:17.543284 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1330157440}
I20240426 09:56:17.575522 10585 leader_election.cc:302] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:17.544668 10081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59746 (request call id 2125) took 1026 ms. Trace:
I20240426 09:56:17.576258 10081 rpcz_store.cc:276] 0426 09:56:16.518241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:16.648523 (+130282us) service_pool.cc:225] Handling call
0426 09:56:17.544531 (+896008us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b5400083c54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:17.544619 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":451749248}
I20240426 09:56:17.575139 10082 rpcz_store.cc:276] 0426 09:56:16.390445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:16.504258 (+113813us) service_pool.cc:225] Handling call
0426 09:56:17.537099 (+1032841us) spinlock_profiling.cc:243] Waited 24.4 ms on lock 0x7b54000da554. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:17.537187 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":56027520}
W20240426 09:56:17.365336 13322 log.cc:924] Time spent T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.894s user 0.005s sys 0.005s
I20240426 09:56:17.555471 9934 rpcz_store.cc:276] 0426 09:56:15.294549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:15.294690 (+ 141us) service_pool.cc:225] Handling call
0426 09:56:15.295005 (+ 315us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:15.295352 (+ 347us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:15.295379 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:16.548907 (+1253528us) spinlock_profiling.cc:243] Waited 1.68 ms on lock 0x7b2c0001c958. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca1628ea1 00007f9ca16384d2 00007f9ca2c81f01 00007f9ca2c4671a 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742
0426 09:56:16.639257 (+ 90350us) log.cc:842] Serialized 26 byte log entry
0426 09:56:16.900017 (+260760us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:56:16.900050 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:16.900112 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:17.494402 (+594290us) spinlock_profiling.cc:243] Waited 3.01 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:56:17.518072 (+ 23670us) raft_consensus.cc:1671] finished
0426 09:56:17.518092 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:17.518226 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":365,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"spinlock_wait_cycles":10784768,"thread_start_us":190647,"threads_started":1,"wal-append.queue_time_us":344080}
I20240426 09:56:17.481798 13386 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:17.321545 13324 raft_consensus.cc:686] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 8 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:17.250461 13345 leader_election.cc:288] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:17.581609 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:17.582243 10014 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:17.582386 10585 leader_election.cc:302] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, 826f03b2fb9c4b9483623b041a1bf80e; no voters:
I20240426 09:56:17.381520 12976 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:17.554994 13327 raft_consensus.cc:482] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:17.553834 13380 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.550829 10011 rpcz_store.cc:276] 0426 09:56:16.494455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.444365 (+949910us) service_pool.cc:225] Handling call
0426 09:56:17.444662 (+ 297us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:17.503473 (+ 58811us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:17.505571 (+ 2098us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:17.544705 13355 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:17.585109 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 334864us
I20240426 09:56:17.509009 13152 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.588166 10583 leader_election.cc:302] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:17.585083 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:17.586274 10613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:17.589057 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:17.590185 10014 raft_consensus.cc:2454] T bbe8813b366b4644872fbec0addf4443 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:56:17.590919 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:17.593698 10081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 636) took 1095 ms. Trace:
I20240426 09:56:17.557657 13378 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:17.610101 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:17.593441 10011 raft_consensus.cc:1261] T 23f4904f98c241de901e223b5ab4725d P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 09:56:17.593046 10082 raft_consensus.cc:1226] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 6. Current term is 7. Ops: []
I20240426 09:56:17.610790 10014 raft_consensus.cc:2379] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f12521b6d51242629cf4297864df00b9 in current term 9: Already voted for candidate 1fee3da01ebc463396397057fdb81a68 in this term.
I20240426 09:56:17.611248 10082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 637) took 1105 ms. Trace:
I20240426 09:56:17.611395 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:17.611660 10082 rpcz_store.cc:276] 0426 09:56:16.505910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.577829 (+1071919us) service_pool.cc:225] Handling call
0426 09:56:17.592894 (+ 15065us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:17.611020 (+ 18126us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:17.611187 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:17.612138 10011 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:56:17.579594 13352 raft_consensus.cc:482] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:17.616874 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5945f6dc42a14b56b9e933f0c1839de1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:17.609315 10081 rpcz_store.cc:276] 0426 09:56:16.498125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.577146 (+1079021us) service_pool.cc:225] Handling call
0426 09:56:17.593270 (+ 16124us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:17.593416 (+ 146us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:17.593439 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:17.593481 (+ 42us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:56:17.593510 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:17.593555 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:17.593650 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:17.648620 10011 raft_consensus.cc:3037] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:17.653380 10088 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 1934) took 2601 ms. Trace:
I20240426 09:56:17.656634 10082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 638) took 1146 ms. Trace:
I20240426 09:56:17.657034 10082 rpcz_store.cc:276] 0426 09:56:16.510448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.612463 (+1102015us) service_pool.cc:225] Handling call
0426 09:56:17.656164 (+ 43701us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:17.656342 (+ 178us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:17.656366 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:17.656404 (+ 38us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:56:17.656440 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:17.656487 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:17.656579 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:17.592079 13230 raft_consensus.cc:2781] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 09:56:17.653797 10088 rpcz_store.cc:276] 0426 09:56:15.052436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:15.052554 (+ 118us) service_pool.cc:225] Handling call
0426 09:56:15.052858 (+ 304us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:15.053179 (+ 321us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:15.053203 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:15.053598 (+ 395us) log.cc:842] Serialized 26 byte log entry
0426 09:56:15.289150 (+235552us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b1800282ca0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fcfdb75 00007f9ca8b37789 00007f9c9fd1b0eb 00007f9c9fd194ef 00007f9c9fd1b652 00007f9ca28c743b 00007f9ca28ccee3 00007f9ca28cd291 00007f9ca2c1a75f
0426 09:56:15.991053 (+701903us) raft_consensus.cc:1631] Marking committed up to 4
0426 09:56:15.991094 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:15.991162 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:17.364844 (+1373682us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b44000232cc. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fc30345 00007f9c9fc30d82 00007f9ca28d9391 00007f9ca28d4c9a 00007f9ca28c8a6e 00007f9ca28c7db6 00007f9ca28c79d8 00007f9ca28d13d1 00007f9ca28d1389 00007f9ca28d1319
0426 09:56:17.653182 (+288338us) raft_consensus.cc:1671] finished
0426 09:56:17.653200 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:17.653321 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":93192,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":7491,"spinlock_wait_cycles":38049024,"thread_start_us":701071,"threads_started":1,"wal-append.queue_time_us":1188944}
I20240426 09:56:17.591843 13285 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:17.597251 13192 raft_consensus.cc:3037] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:17.591984 13058 raft_consensus.cc:2781] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:56:17.591730 13245 raft_consensus.cc:482] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:17.596477 13283 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:17.645076 10615 raft_consensus.cc:1226] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [term 14 FOLLOWER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 13. Current term is 14. Ops: [13.13-13.13]
I20240426 09:56:17.592433 13391 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:17.591648 13229 consensus_queue.cc:1035] T 5e80ebfd190643a99ef02b6ea2876426 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.073s
I20240426 09:56:17.654433 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:17.686866 10011 raft_consensus.cc:2454] T 5945f6dc42a14b56b9e933f0c1839de1 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 7.
I20240426 09:56:17.687806 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:17.688894 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:17.694041 9979 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:17.695439 9979 leader_election.cc:302] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:17.697156 13240 raft_consensus.cc:2781] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:56:17.697530 13240 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:17.697926 13388 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:17.698275 13386 raft_consensus.cc:2726] T 308d6ed4b68346be85caf2871916eab9 P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 09:56:17.592089 13327 raft_consensus.cc:504] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.671389 13230 raft_consensus.cc:686] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [term 9 LEADER]: Becoming Leader. State: Replica: 1fee3da01ebc463396397057fdb81a68, State: Running, Role: LEADER
I20240426 09:56:17.654704 13352 raft_consensus.cc:504] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:17.728951 13352 leader_election.cc:288] T e11b7521039c4feba17dc65ac6c819fb P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:17.675192 13245 raft_consensus.cc:504] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.732597 13245 leader_election.cc:288] T 3c3eaa7da54b489e99a7451385057422 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:17.722664 10082 raft_consensus.cc:1261] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 09:56:17.739153 13194 raft_consensus.cc:482] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:17.739436 10611 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 8.
I20240426 09:56:17.740222 13122 raft_consensus.cc:482] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:17.740685 13122 raft_consensus.cc:504] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:17.743304 13288 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 14 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:17.706138 13325 leader_election.cc:288] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 11 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:17.701642 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:17.706669 9906 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:17.671188 13058 raft_consensus.cc:482] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:17.708005 13390 consensus_queue.cc:1035] T 3dc043e5f7b241afb7119ba8b06ed37f P 826f03b2fb9c4b9483623b041a1bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20240426 09:56:17.697889 13240 raft_consensus.cc:3037] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Advancing to term 4
I20240426 09:56:17.710021 13315 raft_consensus.cc:482] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:17.711081 10618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45570 (request call id 469) took 1228 ms. Trace:
I20240426 09:56:17.646127 13378 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.712437 13380 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:17.698475 13388 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:17.721910 13327 leader_election.cc:288] T 28378f040d6642db8ba229c32bfed21a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:17.691426 13324 consensus_queue.cc:237] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.723862 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:17.681370 13391 raft_consensus.cc:504] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:17.671241 13285 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.677556 13283 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.727157 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1979) took 1234 ms. Trace:
I20240426 09:56:17.739712 13194 raft_consensus.cc:504] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.753527 10618 rpcz_store.cc:276] 0426 09:56:16.482235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:16.482369 (+ 134us) service_pool.cc:225] Handling call
0426 09:56:17.710669 (+1228300us) spinlock_profiling.cc:243] Waited 3.5 ms on lock 0x7b5800047a98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b9cf20 00007f9ca7b88469 00007f9ca7b87f12 00007f9ca126a654
0426 09:56:17.711020 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8052608}
I20240426 09:56:17.765679 13230 consensus_queue.cc:237] T 7b63eef1eb5e4fabb35fb66b0f5a41b4 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:17.753749 13058 raft_consensus.cc:3037] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:56:17.769381 13378 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:17.771953 13152 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:17.733052 13245 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:17.799779 10007 raft_consensus.cc:2454] T 6fde852ed1264555af856522b4abc093 P 826f03b2fb9c4b9483623b041a1bf80e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 3.
I20240426 09:56:17.776422 13330 raft_consensus.cc:2741] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20240426 09:56:17.879751 10010 rpcz_store.cc:276] 0426 09:56:16.492889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:16.495356 (+ 2467us) service_pool.cc:225] Handling call
0426 09:56:16.495559 (+ 203us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:16.505819 (+ 10260us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:16.505846 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:17.545588 (+1039742us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x7b2c00001368. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca1628ea1 00007f9ca16384d2 00007f9ca2c81f01 00007f9ca2c4671a 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742
0426 09:56:17.545937 (+ 349us) log.cc:842] Serialized 26 byte log entry
0426 09:56:17.653450 (+107513us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:56:17.653483 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:17.653548 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:17.726815 (+ 73267us) raft_consensus.cc:1671] finished
0426 09:56:17.726836 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:17.727084 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":56762,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":90,"spinlock_wait_cycles":2332544,"thread_start_us":107223,"threads_started":1,"wal-append.queue_time_us":159580}
I20240426 09:56:17.757330 13259 raft_consensus.cc:3032] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 10 LEADER]: Stepping down as leader of term 10
I20240426 09:56:17.708098 9979 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters: f12521b6d51242629cf4297864df00b9
I20240426 09:56:17.772403 13332 consensus_queue.cc:1035] T 28378f040d6642db8ba229c32bfed21a P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.139s
W20240426 09:56:17.896601 13393 log.cc:924] Time spent T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20240426 09:56:17.925652 13240 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.959535 13240 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 4 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:17.960247 13391 leader_election.cc:288] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
W20240426 09:56:17.933952 10055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 254645us
I20240426 09:56:17.878676 13327 raft_consensus.cc:482] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:17.961033 13327 raft_consensus.cc:504] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.961719 13283 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:17.777825 13203 consensus_queue.cc:1035] T 23f4904f98c241de901e223b5ab4725d P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.141s
I20240426 09:56:17.770857 13315 raft_consensus.cc:504] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:17.917367 13259 raft_consensus.cc:729] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
W20240426 09:56:18.058327 10016 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.002s sys 0.013s
I20240426 09:56:18.128325 10015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 2451) took 1382 ms. Trace:
W20240426 09:56:18.128559 10016 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s user 0.004s sys 0.015s
W20240426 09:56:18.127133 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 376578us
I20240426 09:56:17.935178 13352 raft_consensus.cc:2781] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election won for term 7
W20240426 09:56:18.127732 9979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 419673us
I20240426 09:56:18.130381 9906 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:18.130754 10055 leader_election.cc:302] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:18.131032 10090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 634) took 1652 ms. Trace:
I20240426 09:56:18.074550 13332 raft_consensus.cc:482] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:17.908838 13245 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:18.129109 10015 rpcz_store.cc:276] 0426 09:56:16.746051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.471982 (+725931us) service_pool.cc:225] Handling call
0426 09:56:17.472482 (+ 500us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 09:56:17.476915 (+ 4433us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:17.476942 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 09:56:17.477945 (+ 1003us) log.cc:842] Serialized 532 byte log entry
0426 09:56:17.646105 (+168160us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:56:17.646153 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:17.646229 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:18.128097 (+481868us) raft_consensus.cc:1671] finished
0426 09:56:18.128115 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:18.128272 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3965,"raft.queue_time_us":5420,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"thread_start_us":167283,"threads_started":1,"wal-append.queue_time_us":287286}
I20240426 09:56:18.129482 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:18.165943 10011 raft_consensus.cc:2454] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:56:18.169693 10055 leader_election.cc:302] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:18.170747 10055 leader_election.cc:302] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
W20240426 09:56:18.195374 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 65072.7us
W20240426 09:56:18.205962 10054 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.003s sys 0.017s
W20240426 09:56:18.206339 10054 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.513s user 0.003s sys 0.017s
I20240426 09:56:18.209772 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:18.211997 10011 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Advancing to term 10
W20240426 09:56:18.058915 13242 consensus_meta.cc:287] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.507s user 0.007s sys 0.000s
I20240426 09:56:18.215234 13327 leader_election.cc:288] T 693cab52ca1f45cab4af9282a6614f90 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:17.984377 13240 raft_consensus.cc:482] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:18.088889 13330 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:18.240715 10084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 1961) took 1844 ms. Trace:
I20240426 09:56:18.127449 13302 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:18.208273 13380 raft_consensus.cc:2781] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:18.253939 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:18.256511 9906 leader_election.cc:302] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters: d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:18.201702 13352 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting leader election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:18.208240 13117 raft_consensus.cc:2781] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20240426 09:56:18.216171 13123 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.030s sys 0.004s
I20240426 09:56:18.208141 13378 raft_consensus.cc:2781] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 09:56:18.122196 13283 consensus_queue.cc:1046] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.138s
I20240426 09:56:18.209966 13332 raft_consensus.cc:504] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:18.286600 10013 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.712s user 0.007s sys 0.045s
W20240426 09:56:18.424352 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 167936us
W20240426 09:56:18.432358 10013 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.858s user 0.016s sys 0.048s
I20240426 09:56:18.297310 10015 raft_consensus.cc:1226] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 8. Current term is 9. Ops: []
I20240426 09:56:18.209425 13313 raft_consensus.cc:2781] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:18.425323 10618 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:18.428092 13152 raft_consensus.cc:2781] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election won for term 7
W20240426 09:56:18.245118 13257 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.009s sys 0.014s
I20240426 09:56:18.256986 10084 rpcz_store.cc:276] 0426 09:56:16.396764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.249068 (+852304us) service_pool.cc:225] Handling call
0426 09:56:17.320057 (+ 70989us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:17.320370 (+ 313us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 09:56:17.320394 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:17.320763 (+ 369us) log.cc:842] Serialized 26 byte log entry
0426 09:56:17.580275 (+259512us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:56:17.580343 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:17.580407 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:18.240502 (+660095us) raft_consensus.cc:1671] finished
0426 09:56:18.240531 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:18.240657 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":724,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":34630,"spinlock_wait_cycles":2432,"thread_start_us":161343,"threads_started":1,"wal-append.queue_time_us":321593}
I20240426 09:56:18.438338 9906 leader_election.cc:302] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:56:18.424865 13123 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.726s user 0.033s sys 0.011s
I20240426 09:56:18.428449 13314 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
W20240426 09:56:18.436702 13248 consensus_meta.cc:287] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 0.862s user 0.008s sys 0.000s
W20240426 09:56:18.435251 13257 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.715s user 0.010s sys 0.014s
W20240426 09:56:18.266450 13398 log.cc:924] Time spent T 2e197425a8c74512b4dac9d31a3ce0a5 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20240426 09:56:18.440328 13402 log.cc:924] Time spent T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.455s user 0.001s sys 0.000s
I20240426 09:56:18.368387 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:18.423601 13117 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:18.331537 13229 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:18.475196 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b63eef1eb5e4fabb35fb66b0f5a41b4" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:18.428411 13352 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:18.433552 13380 raft_consensus.cc:482] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:18.461452 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472a3b5debc94412b63080f474ae4fed" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:18.462105 10083 raft_consensus.cc:2454] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 9.
I20240426 09:56:18.428218 13378 raft_consensus.cc:482] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:18.462716 10011 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 10.
I20240426 09:56:18.435420 13313 raft_consensus.cc:482] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:18.440570 13285 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:18.467306 10007 raft_consensus.cc:1226] T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 LEADER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 6. Current term is 7. Ops: []
I20240426 09:56:18.471688 13122 leader_election.cc:288] T c6ed47e1980f4a14a46cc61ea156eb53 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:18.473081 10619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51158 (request call id 525) took 1387 ms. Trace:
W20240426 09:56:18.474819 10612 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51158 (request call id 519) took 1928 ms (client timeout 1699 ms). Trace:
I20240426 09:56:18.262348 13240 raft_consensus.cc:504] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:18.459110 13192 consensus_meta.cc:287] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68: Time spent flushing consensus metadata: real 0.782s user 0.004s sys 0.004s
I20240426 09:56:18.509491 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc043e5f7b241afb7119ba8b06ed37f" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 14 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:18.478327 13257 raft_consensus.cc:482] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:18.513881 10084 raft_consensus.cc:3037] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 13 FOLLOWER]: Advancing to term 14
W20240426 09:56:18.512353 10612 rpcz_store.cc:269] 0426 09:56:16.546526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:16.546704 (+ 178us) service_pool.cc:225] Handling call
0426 09:56:18.474599 (+1927895us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b5400504454. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:18.474739 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":788024704}
I20240426 09:56:18.283017 13302 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:18.511631 10619 rpcz_store.cc:276] 0426 09:56:17.085845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.085962 (+ 117us) service_pool.cc:225] Handling call
0426 09:56:17.086234 (+ 272us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:17.521536 (+435302us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:18.472833 (+951297us) spinlock_profiling.cc:243] Waited 435 ms on lock 0x7b54007805d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:18.473017 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":999947648}
I20240426 09:56:18.488343 10616 raft_consensus.cc:2454] T a7ef86a50a514e8ba09c76ca674b2100 P 1fee3da01ebc463396397057fdb81a68 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:56:18.275643 13259 consensus_queue.cc:260] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:18.288659 13330 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:18.452550 13314 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:18.489998 13117 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:18.515733 10618 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P 1fee3da01ebc463396397057fdb81a68 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 11.
I20240426 09:56:18.508919 13378 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:18.493952 13336 raft_consensus.cc:482] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:18.508261 13313 raft_consensus.cc:3037] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:18.507457 13380 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:18.514379 13388 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:18.524345 13192 raft_consensus.cc:2726] T 21e4aa688b194564bef9b4398f572caf P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 09:56:18.493256 13229 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:18.548825 13330 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:18.551096 13314 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:18.535704 13259 raft_consensus.cc:3037] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:18.554334 13380 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:18.534986 10617 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51158 (request call id 523) took 1635 ms (client timeout 1666 ms). Trace:
W20240426 09:56:18.545845 13296 log.cc:924] Time spent T 00000000000000000000000000000000 P d0e04ed59b99401d8e95cbeab83582c3: Append to log took a long time: real 3.111s user 0.026s sys 0.029s
I20240426 09:56:18.567044 10084 raft_consensus.cc:2454] T 3dc043e5f7b241afb7119ba8b06ed37f P d3a480e831d649139ff016fb68aee3ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 14.
I20240426 09:56:18.524195 13257 raft_consensus.cc:504] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
W20240426 09:56:18.572399 10010 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.002s sys 0.022s
W20240426 09:56:18.572816 10010 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.606s user 0.002s sys 0.023s
I20240426 09:56:18.557431 13315 leader_election.cc:288] T 6cfada00f42a411f813bfc35d6c9ec9d P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:18.541219 13336 raft_consensus.cc:504] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:18.560715 13245 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:18.602703 13388 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:18.611246 13388 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:18.613188 13388 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:18.614007 13411 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:18.614548 13411 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:18.616487 13411 leader_election.cc:288] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:18.616945 13411 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:18.602936 13386 raft_consensus.cc:482] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:18.618434 13386 raft_consensus.cc:504] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:18.620306 13386 leader_election.cc:288] T 9da46d724daf4bbf9c49a393e3579156 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:18.620532 13289 consensus_meta.cc:287] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 1.091s user 0.009s sys 0.001s
I20240426 09:56:18.620975 13289 raft_consensus.cc:2726] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 09:56:18.621490 13223 raft_consensus.cc:2781] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 09:56:18.596153 13152 raft_consensus.cc:686] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [term 7 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:18.622347 10086 raft_consensus.cc:1226] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:18.622079 13223 raft_consensus.cc:482] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
W20240426 09:56:18.559536 10617 rpcz_store.cc:269] 0426 09:56:16.894649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:16.894997 (+ 348us) service_pool.cc:225] Handling call
0426 09:56:18.475052 (+1580055us) spinlock_profiling.cc:243] Waited 2.51 ms on lock 0x7b5800047a98. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9c9fb62811 00007f9c9fce4071 00007f9c9fce3f22 00007f9ca8bd66d9 00007f9ca7b13270 00007f9ca7bd927b 00007f9ca7bd9301 00007f9ca7b88e20 00007f9ca7b88704 00007f9ca126cbd4 00007f9ca126cb62
0426 09:56:18.530033 (+ 54981us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5778816}
W20240426 09:56:18.603610 12983 consensus_meta.cc:287] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee: Time spent flushing consensus metadata: real 1.106s user 0.009s sys 0.002s
W20240426 09:56:18.602882 13350 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.124s user 0.043s sys 0.000s
I20240426 09:56:18.621539 13338 raft_consensus.cc:3032] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 09:56:18.617173 12917 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:18.617401 13411 raft_consensus.cc:504] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:18.639768 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40928 (request call id 673) took 1338 ms. Trace:
I20240426 09:56:18.655616 9937 rpcz_store.cc:276] 0426 09:56:17.301058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.301197 (+ 139us) service_pool.cc:225] Handling call
0426 09:56:18.639593 (+1338396us) spinlock_profiling.cc:243] Waited 37.8 ms on lock 0x7b5400375fd4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:18.639713 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":86921600}
W20240426 09:56:18.641842 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 454773us
W20240426 09:56:18.640414 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 202125us
I20240426 09:56:18.605675 13332 leader_election.cc:288] T 09b98bf14ce848eca1c7ca1840a516b1 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:18.622557 13302 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:18.633562 13386 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:18.634462 13223 raft_consensus.cc:3037] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 3 FOLLOWER]: Advancing to term 4
W20240426 09:56:18.638170 13350 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.158s user 0.043s sys 0.001s
W20240426 09:56:18.640820 13058 consensus_meta.cc:287] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68: Time spent flushing consensus metadata: real 0.725s user 0.009s sys 0.001s
I20240426 09:56:18.681802 13332 raft_consensus.cc:482] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:18.671999 12917 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:18.638938 13338 raft_consensus.cc:729] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:18.641433 13229 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:18.687520 9905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 443591us
I20240426 09:56:18.688117 10012 raft_consensus.cc:1226] T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e [term 8 LEADER]: Rejecting Update request from peer 1fee3da01ebc463396397057fdb81a68 for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:18.684175 13058 raft_consensus.cc:504] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:18.677966 9906 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
W20240426 09:56:18.712218 10088 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.806s user 0.009s sys 0.010s
W20240426 09:56:18.692929 13408 log.cc:924] Time spent T 67dfdb63e28a460486dfe163f03bf820 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.163s user 0.000s sys 0.001s
W20240426 09:56:18.712827 10088 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.806s user 0.009s sys 0.010s
I20240426 09:56:18.637378 12983 raft_consensus.cc:504] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:18.683121 13386 raft_consensus.cc:504] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:18.727732 10087 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 7.
W20240426 09:56:18.729228 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 51336.8us
I20240426 09:56:18.730460 9906 leader_election.cc:302] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:18.693655 13332 raft_consensus.cc:504] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:18.740568 10620 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45570 (request call id 470) took 2231 ms (client timeout 2031 ms). Trace:
W20240426 09:56:18.740967 10620 rpcz_store.cc:269] 0426 09:56:16.509273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:16.509970 (+ 697us) service_pool.cc:225] Handling call
0426 09:56:18.740379 (+2230409us) spinlock_profiling.cc:243] Waited 865 ms on lock 0x7b5400601654. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:18.740500 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1988472576}
I20240426 09:56:18.725751 13324 raft_consensus.cc:482] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:18.731326 13380 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:18.736106 13203 raft_consensus.cc:2781] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 09:56:18.734781 13421 raft_consensus.cc:2781] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:56:18.712357 13152 consensus_queue.cc:237] T 5945f6dc42a14b56b9e933f0c1839de1 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:18.735993 13412 raft_consensus.cc:482] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:18.731572 13240 leader_election.cc:288] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:18.748490 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 2455) took 1263 ms. Trace:
I20240426 09:56:18.753984 13421 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:56:18.762811 10010 rpcz_store.cc:276] 0426 09:56:17.484653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.913403 (+428750us) service_pool.cc:225] Handling call
0426 09:56:17.913699 (+ 296us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:17.913897 (+ 198us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 09:56:18.592201 (+678304us) log.cc:842] Serialized 14 byte log entry
0426 09:56:18.747441 (+155240us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:18.747524 (+ 83us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:56:18.747598 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:18.747715 (+ 117us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:18.747909 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5691,"prepare.queue_time_us":819126,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":6610,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":9851,"thread_start_us":752872,"threads_started":2}
I20240426 09:56:18.742652 13417 consensus_queue.cc:1046] T 693cab52ca1f45cab4af9282a6614f90 P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
W20240426 09:56:18.766649 13410 log.cc:924] Time spent T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.165s user 0.000s sys 0.001s
I20240426 09:56:18.785256 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b98bf14ce848eca1c7ca1840a516b1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:18.785976 13058 leader_election.cc:288] T eb5d051be8c743a18cd550e145b6564e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 3 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
W20240426 09:56:18.786824 13419 log.cc:924] Time spent T 007f1451130c4dfcbd86145d56a6f67e P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20240426 09:56:18.788018 13378 raft_consensus.cc:504] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:18.789522 13313 raft_consensus.cc:504] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:18.747407 13324 raft_consensus.cc:504] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:18.773010 13421 raft_consensus.cc:3037] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:18.801051 13411 leader_election.cc:288] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
W20240426 09:56:18.805588 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 597668us
W20240426 09:56:18.806697 9981 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.004s sys 0.012s
I20240426 09:56:18.807010 13248 raft_consensus.cc:3032] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Stepping down as leader of term 5
I20240426 09:56:18.809008 10088 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 1965) took 1876 ms. Trace:
I20240426 09:56:18.809425 13380 raft_consensus.cc:482] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:18.810389 13223 raft_consensus.cc:504] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:18.811349 10089 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59746 (request call id 2119) took 2633 ms. Trace:
I20240426 09:56:18.815806 9939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 2469) took 1938 ms. Trace:
I20240426 09:56:18.816555 13330 raft_consensus.cc:482] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:56:18.816810 9941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47248 (request call id 2468) took 1942 ms (client timeout 2031 ms). Trace:
I20240426 09:56:18.818094 10014 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 770) took 1908 ms. Trace:
W20240426 09:56:18.821154 13422 log.cc:924] Time spent T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
W20240426 09:56:18.821544 13407 log.cc:924] Time spent T 7a3634dc56b34299b112ff8555d47c1e P f12521b6d51242629cf4297864df00b9: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20240426 09:56:18.841042 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574d769068f477e859574c959d5880e" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:18.737416 13338 consensus_queue.cc:260] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:18.773216 13257 leader_election.cc:288] T 2f96620cc62e4bf1beefad6b59ca3968 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:18.822223 12917 leader_election.cc:288] T ac0131f6d1d54716ab3324104a18c9c3 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:18.893122 10010 raft_consensus.cc:2454] T 8574d769068f477e859574c959d5880e P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 9.
I20240426 09:56:18.887861 13259 raft_consensus.cc:482] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:18.894557 10010 raft_consensus.cc:1261] T ab7abcb2e80641eca676add7616cd966 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 09:56:18.894817 13244 raft_consensus.cc:3037] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:18.895148 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 2003) took 1222 ms. Trace:
I20240426 09:56:18.895481 10010 rpcz_store.cc:276] 0426 09:56:17.672424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:18.893979 (+1221555us) service_pool.cc:225] Handling call
0426 09:56:18.894253 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:18.894921 (+ 668us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:18.895096 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:18.896590 10014 rpcz_store.cc:276] 0426 09:56:16.910074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.612299 (+702225us) service_pool.cc:225] Handling call
0426 09:56:17.612554 (+ 255us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:17.729534 (+116980us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:17.729564 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:17.729603 (+ 39us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:56:17.729633 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:18.817854 (+1088221us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b540016c454. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:18.817900 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:18.818036 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":268654464}
I20240426 09:56:18.898092 10014 raft_consensus.cc:1226] T 5e80ebfd190643a99ef02b6ea2876426 P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Rejecting Update request from peer f12521b6d51242629cf4297864df00b9 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20240426 09:56:18.898627 10014 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 2005) took 1221 ms. Trace:
I20240426 09:56:18.899570 13338 raft_consensus.cc:3037] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:18.900776 13332 leader_election.cc:288] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:18.901110 10085 raft_consensus.cc:3037] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Advancing to term 7
I20240426 09:56:18.902314 13378 leader_election.cc:288] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 11 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:18.903934 10013 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 2452) took 2157 ms. Trace:
I20240426 09:56:18.765213 13412 raft_consensus.cc:504] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:18.904119 10086 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48202 (request call id 635) took 2409 ms. Trace:
W20240426 09:56:18.904706 9981 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.765s user 0.004s sys 0.019s
W20240426 09:56:18.905431 9941 rpcz_store.cc:269] 0426 09:56:16.874084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:16.895755 (+ 21671us) service_pool.cc:225] Handling call
0426 09:56:18.816642 (+1920887us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b54003ee254. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:18.816754 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":512736640}
I20240426 09:56:18.898957 10014 rpcz_store.cc:276] 0426 09:56:17.676982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:18.897591 (+1220609us) service_pool.cc:225] Handling call
0426 09:56:18.897897 (+ 306us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:18.898416 (+ 519us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:18.898579 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:18.909080 10086 rpcz_store.cc:276] 0426 09:56:16.494366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.552596 (+1058230us) service_pool.cc:225] Handling call
0426 09:56:17.552862 (+ 266us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:18.629779 (+1076917us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:18.894681 (+264902us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400125054. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:18.904062 (+ 9381us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2459217280}
I20240426 09:56:18.910291 9906 leader_election.cc:302] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:18.911172 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.908085 10010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:18.911744 10086 raft_consensus.cc:2379] T 9c2caa6ec04942588af2288b0ac9ef6b P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 11: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:18.166893 10090 rpcz_store.cc:276] 0426 09:56:16.478936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.523857 (+1044921us) service_pool.cc:225] Handling call
0426 09:56:17.524154 (+ 297us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:17.524480 (+ 326us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:17.524506 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:17.524895 (+ 389us) log.cc:842] Serialized 26 byte log entry
0426 09:56:17.655308 (+130413us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:56:18.129769 (+474461us) log.cc:842] Serialized 14 byte log entry
0426 09:56:18.130733 (+ 964us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:18.130820 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:18.130860 (+ 40us) raft_consensus.cc:1671] finished
0426 09:56:18.130872 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:18.130980 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1055,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":28127,"thread_start_us":550697,"threads_started":2,"wal-append.queue_time_us":155833}
I20240426 09:56:18.904295 10013 rpcz_store.cc:276] 0426 09:56:16.746323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.503013 (+756690us) service_pool.cc:225] Handling call
0426 09:56:17.503336 (+ 323us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:17.503632 (+ 296us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:17.503659 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:17.513368 (+ 9709us) log.cc:842] Serialized 26 byte log entry
0426 09:56:18.432881 (+919513us) raft_consensus.cc:1631] Marking committed up to 4
0426 09:56:18.432927 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:18.433004 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:18.903722 (+470718us) raft_consensus.cc:1671] finished
0426 09:56:18.903742 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:18.903880 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":830,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":10828,"thread_start_us":858025,"threads_started":1,"wal-append.queue_time_us":995026}
I20240426 09:56:18.912379 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.913867 10090 raft_consensus.cc:2396] T 9da46d724daf4bbf9c49a393e3579156 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20240426 09:56:18.904999 10088 rpcz_store.cc:276] 0426 09:56:16.932293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.662795 (+730502us) service_pool.cc:225] Handling call
0426 09:56:17.675892 (+ 13097us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:17.676099 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 09:56:17.906424 (+230325us) log.cc:842] Serialized 14 byte log entry
0426 09:56:18.808683 (+902259us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:18.808731 (+ 48us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:56:18.808778 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:18.808841 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:18.808960 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":92100,"prepare.queue_time_us":672206,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":24,"raft.queue_time_us":93325,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"thread_start_us":1033107,"threads_started":2,"wal-append.queue_time_us":844265}
I20240426 09:56:18.910835 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:18.915617 10010 raft_consensus.cc:1203] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Deduplicated request from leader. Original: 5.3->[6.4-6.4] Dedup: 6.4->[]
I20240426 09:56:18.916195 10010 raft_consensus.cc:1226] T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Rejecting Update request from peer d3a480e831d649139ff016fb68aee3ee for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240426 09:56:18.916772 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34378 (request call id 2461) took 1237 ms. Trace:
I20240426 09:56:18.904645 10089 rpcz_store.cc:276] 0426 09:56:16.177841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.086679 (+908838us) service_pool.cc:225] Handling call
0426 09:56:17.088165 (+ 1486us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:17.088458 (+ 293us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:17.088485 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:17.093711 (+ 5226us) log.cc:842] Serialized 26 byte log entry
0426 09:56:18.676271 (+1582560us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b180048e8c0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9ca28d56ac 00007f9c9fcfe20c 00007f9ca8b37789 00007f9c9fd1b0eb 00007f9c9fd194ef 00007f9c9fd1b652 00007f9ca28c743b 00007f9ca28ccee3 00007f9ca28cd291
0426 09:56:18.677960 (+ 1689us) raft_consensus.cc:1631] Marking committed up to 10
0426 09:56:18.683869 (+ 5909us) log.cc:842] Serialized 14 byte log entry
0426 09:56:18.811014 (+127145us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:18.811084 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:18.811122 (+ 38us) raft_consensus.cc:1671] finished
0426 09:56:18.811196 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:18.811303 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57629,"prepare.queue_time_us":8910,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":1792,"raft.queue_time_us":138819,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":8861,"spinlock_wait_cycles":37688064,"thread_start_us":79786,"threads_started":2,"wal-append.queue_time_us":224343,"wal-append.run_cpu_time_us":2179,"wal-append.run_wall_time_us":1046608}
I20240426 09:56:18.917171 10010 rpcz_store.cc:276] 0426 09:56:17.679019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:18.913893 (+1234874us) service_pool.cc:225] Handling call
0426 09:56:18.914196 (+ 303us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:18.916564 (+ 2368us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:18.916727 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:18.917802 9939 rpcz_store.cc:276] 0426 09:56:16.877486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:16.877594 (+ 108us) service_pool.cc:225] Handling call
0426 09:56:16.902522 (+ 24928us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:17.165416 (+262894us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:18.815557 (+1650141us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b54003769d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:18.815743 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":512877312}
I20240426 09:56:18.918089 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
W20240426 09:56:18.890184 13424 log.cc:924] Time spent T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20240426 09:56:18.921026 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.921797 10089 raft_consensus.cc:2379] T 2f96620cc62e4bf1beefad6b59ca3968 P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e in current term 11: Already voted for candidate d3a480e831d649139ff016fb68aee3ee in this term.
I20240426 09:56:18.927628 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.928974 10086 raft_consensus.cc:2454] T cfa61a21874f49688b31ddd60646a658 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 8.
I20240426 09:56:18.929111 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.930039 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.929723 10090 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 6.
I20240426 09:56:18.932310 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.933898 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.934999 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.935546 10089 raft_consensus.cc:2454] T d9f79bd8994f4b229ca8da8638bb6d5a P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 5.
I20240426 09:56:18.936329 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.936897 10089 raft_consensus.cc:2454] T 253cc496ee9644ffba6201074808dbf0 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 7.
I20240426 09:56:18.937664 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.904156 12983 leader_election.cc:288] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:18.908556 13248 raft_consensus.cc:729] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:18.939513 9979 leader_election.cc:302] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:18.939426 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:18.898980 13347 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:18.940958 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.938562 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.942955 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.946664 10086 raft_consensus.cc:2396] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 09:56:18.947502 9979 leader_election.cc:302] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:18.948160 9979 leader_election.cc:302] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:18.927780 13306 raft_consensus.cc:482] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:18.956310 13306 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:18.958119 13306 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:18.958647 13386 leader_election.cc:288] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:18.928408 13305 raft_consensus.cc:482] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:18.921334 13321 raft_consensus.cc:482] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:56:18.949486 10054 consensus_peers.cc:485] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee -> Peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Couldn't send request to peer 1fee3da01ebc463396397057fdb81a68. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240426 09:56:18.945446 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:18.950712 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6c52e0c924ba3b3fca5f9586814a7" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:18.954107 13223 leader_election.cc:288] T 6fde852ed1264555af856522b4abc093 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 4 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:18.964222 13324 leader_election.cc:288] T fb385fda44e94153ae3cd9ca21240939 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:18.966172 13412 leader_election.cc:288] T be50ac00c79946d89ecaf230084a401d P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:18.967079 12976 raft_consensus.cc:2781] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 09:56:18.927724 13242 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:18.912189 13380 raft_consensus.cc:504] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:18.981933 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
W20240426 09:56:18.981482 9906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.130:44411, user_credentials={real_user=slave}} blocked reactor thread for 60886.8us
I20240426 09:56:18.980827 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:18.979498 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.909955 13259 raft_consensus.cc:504] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:18.939978 13152 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:18.953912 13391 raft_consensus.cc:2932] T 1b76245317c24e129e7f16cdb4d99007 P d3a480e831d649139ff016fb68aee3ee [term 11 LEADER]: Committing config change with OpId 9.7: config changed from index -1 to 7, NON_VOTER 1fee3da01ebc463396397057fdb81a68 (127.8.141.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } } }
I20240426 09:56:18.858865 13203 raft_consensus.cc:686] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [term 10 LEADER]: Becoming Leader. State: Replica: f12521b6d51242629cf4297864df00b9, State: Running, Role: LEADER
I20240426 09:56:18.983079 10086 raft_consensus.cc:2454] T 17e4aa133eb141699b9a5e2f0b4767d6 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 10.
I20240426 09:56:18.983064 10014 raft_consensus.cc:2454] T a202d889782b4735855ec95f82e22ac8 P 826f03b2fb9c4b9483623b041a1bf80e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
W20240426 09:56:18.978541 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 69486us
I20240426 09:56:18.910063 13330 raft_consensus.cc:504] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:19.009486 13117 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:19.009923 13437 consensus_queue.cc:1035] T ab7abcb2e80641eca676add7616cd966 P f12521b6d51242629cf4297864df00b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240426 09:56:19.012936 13117 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:19.013659 13203 consensus_queue.cc:237] T bc3e15fcf67145f38946b62f0a5e9149 P f12521b6d51242629cf4297864df00b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:19.010038 13242 raft_consensus.cc:504] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:19.014588 10084 raft_consensus.cc:1261] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20240426 09:56:19.014972 10084 pending_rounds.cc:76] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee: Aborting all ops after (but not including) 5
I20240426 09:56:19.015172 10084 pending_rounds.cc:97] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20240426 09:56:19.015388 10084 raft_consensus.cc:2866] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 09:56:19.016680 10088 raft_consensus.cc:3037] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:18.962196 13411 raft_consensus.cc:2781] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 09:56:18.968755 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1ede5c9b614a0f8b7adb20a7cfbbb1" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:19.020951 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76245317c24e129e7f16cdb4d99007" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.987412 9979 leader_election.cc:302] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e; no voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9
I20240426 09:56:19.005570 10055 leader_election.cc:302] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:18.967649 13347 raft_consensus.cc:482] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:19.009102 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:19.004695 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee"
I20240426 09:56:18.979947 12917 raft_consensus.cc:482] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:18.984051 12976 raft_consensus.cc:482] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:19.019701 10089 raft_consensus.cc:2454] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 9.
I20240426 09:56:19.028579 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:19.027223 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e4aa688b194564bef9b4398f572caf" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.962543 13351 raft_consensus.cc:2781] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:19.023756 13386 raft_consensus.cc:482] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:19.015929 13242 leader_election.cc:288] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:19.001996 13375 raft_consensus.cc:482] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:19.032366 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde852ed1264555af856522b4abc093" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:19.016204 13366 raft_consensus.cc:482] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:19.035395 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:18.984408 13412 raft_consensus.cc:482] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:19.038106 9979 leader_election.cc:302] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826f03b2fb9c4b9483623b041a1bf80e, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:19.003787 10054 leader_election.cc:302] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:18.979627 13321 raft_consensus.cc:504] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:18.966183 13248 consensus_queue.cc:260] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:19.031014 13257 raft_consensus.cc:2726] T 72a6c52e0c924ba3b3fca5f9586814a7 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 09:56:19.031105 12976 raft_consensus.cc:3037] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 9 FOLLOWER]: Advancing to term 10
I20240426 09:56:19.048877 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.033288 13351 raft_consensus.cc:482] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:19.040593 10088 raft_consensus.cc:2454] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 10.
I20240426 09:56:19.022325 13411 raft_consensus.cc:482] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:19.044128 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c21d6dedd1441bd977f09cdcf41d436" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.004482 13152 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:19.044807 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:19.046712 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28378f040d6642db8ba229c32bfed21a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.047120 10054 leader_election.cc:302] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:19.046484 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.048381 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eaa7da54b489e99a7451385057422" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.045599 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.041456 13193 raft_consensus.cc:3037] T bbe8813b366b4644872fbec0addf4443 P 1fee3da01ebc463396397057fdb81a68 [term 4 FOLLOWER]: Advancing to term 5
I20240426 09:56:19.054092 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.087669 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e7dd8bbe64a3eb2174cd4a218d51c" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:19.086184 9937 raft_consensus.cc:2454] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:56:19.089211 9932 raft_consensus.cc:3037] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:19.042829 13313 leader_election.cc:288] T bbe8813b366b4644872fbec0addf4443 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 6 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:19.025929 13347 raft_consensus.cc:504] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:19.030472 12917 raft_consensus.cc:504] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: NON_VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: true } }
I20240426 09:56:19.052237 10014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:18.979691 13305 raft_consensus.cc:504] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:19.089054 9935 raft_consensus.cc:2454] T 3c3eaa7da54b489e99a7451385057422 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:19.083788 13289 raft_consensus.cc:482] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:19.094002 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bebc132e4141bdbab199326986516e" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.102632 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:19.048712 13242 raft_consensus.cc:2781] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Leader election won for term 11
I20240426 09:56:19.111585 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc982aface54033a68641d8d809fd3b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.095455 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.112393 9933 raft_consensus.cc:2454] T 35fd22bc1cf64e38840dcfb850cfe090 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 5.
I20240426 09:56:19.096578 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
W20240426 09:56:19.097081 9982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 47206us
I20240426 09:56:19.044420 13117 raft_consensus.cc:482] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:19.046725 13227 raft_consensus.cc:2781] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 09:56:19.044211 13412 raft_consensus.cc:504] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:19.087213 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.098027 12983 raft_consensus.cc:2781] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:19.083652 13351 raft_consensus.cc:3037] T d9f79bd8994f4b229ca8da8638bb6d5a P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:19.089679 13248 raft_consensus.cc:3037] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:19.094009 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.133049 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6151686b1a24447880e49b7d7395fcc1" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:19.142860 9936 raft_consensus.cc:2379] T 6151686b1a24447880e49b7d7395fcc1 P f12521b6d51242629cf4297864df00b9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 7: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:19.037042 13386 raft_consensus.cc:504] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:19.098650 13315 raft_consensus.cc:482] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:19.087898 13411 raft_consensus.cc:3037] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Advancing to term 9
I20240426 09:56:19.101135 10616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe8813b366b4644872fbec0addf4443" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68"
I20240426 09:56:19.043115 13366 raft_consensus.cc:504] T 23f4904f98c241de901e223b5ab4725d P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:19.050387 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.107167 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.050029 13119 raft_consensus.cc:2781] T 253cc496ee9644ffba6201074808dbf0 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:19.113106 9934 raft_consensus.cc:2454] T 5fc982aface54033a68641d8d809fd3b P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:19.170746 9940 raft_consensus.cc:2454] T 6cfada00f42a411f813bfc35d6c9ec9d P f12521b6d51242629cf4297864df00b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 4.
I20240426 09:56:19.093161 13306 raft_consensus.cc:482] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 826f03b2fb9c4b9483623b041a1bf80e)
I20240426 09:56:19.041023 13375 raft_consensus.cc:504] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:19.141559 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:19.106427 9932 raft_consensus.cc:2454] T c75e7dd8bbe64a3eb2174cd4a218d51c P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 11.
I20240426 09:56:19.171885 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:19.113992 13227 raft_consensus.cc:482] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:19.177145 10085 raft_consensus.cc:1261] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Refusing update from remote peer f12521b6d51242629cf4297864df00b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 09:56:19.199128 10014 raft_consensus.cc:2454] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 6.
I20240426 09:56:19.119041 13242 raft_consensus.cc:686] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [term 11 LEADER]: Becoming Leader. State: Replica: d3a480e831d649139ff016fb68aee3ee, State: Running, Role: LEADER
I20240426 09:56:19.143642 13330 leader_election.cc:288] T 3dc043e5f7b241afb7119ba8b06ed37f P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:19.229540 13242 consensus_queue.cc:237] T c75e7dd8bbe64a3eb2174cd4a218d51c P d3a480e831d649139ff016fb68aee3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:19.231519 13421 raft_consensus.cc:504] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:19.233398 13421 leader_election.cc:288] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 5 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:19.113677 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1fee3da01ebc463396397057fdb81a68" is_pre_election: true
I20240426 09:56:19.234560 10617 raft_consensus.cc:2454] T cf8634deb0a5425db9c47915d49386d5 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:56:19.190382 9939 raft_consensus.cc:2379] T a3af5b1ab22b40b2abe93efd45bb7676 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 8: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:19.235810 10620 raft_consensus.cc:2454] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 7.
I20240426 09:56:19.236650 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8634deb0a5425db9c47915d49386d5" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.237840 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.237856 10013 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 2006) took 1556 ms. Trace:
I20240426 09:56:19.238919 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.127058 13276 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:19.128147 13152 leader_election.cc:288] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:19.212249 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 1937) took 3994 ms. Trace:
W20240426 09:56:19.214862 13046 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
W20240426 09:56:19.173578 10611 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51158 (request call id 515) took 3764 ms (client timeout 1961 ms). Trace:
W20240426 09:56:19.216006 13352 consensus_meta.cc:287] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e: Time spent flushing consensus metadata: real 0.688s user 0.009s sys 0.000s
W20240426 09:56:19.216861 13443 log.cc:924] Time spent T 7a3634dc56b34299b112ff8555d47c1e P 826f03b2fb9c4b9483623b041a1bf80e: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20240426 09:56:19.217736 13347 leader_election.cc:288] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:19.209686 13373 raft_consensus.cc:482] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:19.219899 13321 leader_election.cc:288] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:19.227159 13227 raft_consensus.cc:3037] T 17e4aa133eb141699b9a5e2f0b4767d6 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:19.228015 12917 leader_election.cc:288] T 1b76245317c24e129e7f16cdb4d99007 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:19.228431 13386 leader_election.cc:288] T 2e197425a8c74512b4dac9d31a3ce0a5 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:19.116520 13412 leader_election.cc:288] T 51125fbde06b4a588fb6171dec56a605 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:19.112921 13289 raft_consensus.cc:504] T 67dfdb63e28a460486dfe163f03bf820 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:19.113821 13117 raft_consensus.cc:504] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:19.443672 10086 raft_consensus.cc:2454] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 4.
I20240426 09:56:19.239063 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.454015 10085 rpcz_store.cc:276] 0426 09:56:15.217657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:15.219790 (+ 2133us) service_pool.cc:225] Handling call
0426 09:56:15.220002 (+ 212us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:17.085635 (+1865633us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b54003ad020. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcbc70 00007f9ca2bd68b3 00007f9ca2c48490 00007f9ca2c49501 00007f9ca2c4606f 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c
0426 09:56:18.900721 (+1815086us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b5800755fb0. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2bc20e3 00007f9ca2c25c4a 00007f9ca2c3d53f 00007f9ca2c3ecde 00007f9ca2c48e19 00007f9ca2c496e6 00007f9ca2c4606f 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654
0426 09:56:19.211807 (+311086us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:19.212193 (+ 386us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1084722304}
W20240426 09:56:19.335778 10087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 640) took 1861 ms (client timeout 1949 ms). Trace:
I20240426 09:56:19.443554 10054 leader_election.cc:302] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:19.239466 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951e4c48fcb642e58ab79652d3f86261" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:19.457454 10085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69700f6b120b4f4ebc0b43b14166f4c9" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:19.316370 10090 raft_consensus.cc:2454] T be50ac00c79946d89ecaf230084a401d P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:56:19.445207 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.456694 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51125fbde06b4a588fb6171dec56a605" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:19.459466 9939 raft_consensus.cc:2379] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:19.460093 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:19.460554 10086 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 8.
I20240426 09:56:19.461179 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b7521039c4feba17dc65ac6c819fb" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:19.461598 10086 raft_consensus.cc:2454] T e11b7521039c4feba17dc65ac6c819fb P d3a480e831d649139ff016fb68aee3ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
W20240426 09:56:19.457123 10087 rpcz_store.cc:269] 0426 09:56:17.474058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.507796 (+ 33738us) service_pool.cc:225] Handling call
0426 09:56:19.335568 (+1827772us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400083754. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:19.335713 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2803515136}
I20240426 09:56:19.238437 10013 rpcz_store.cc:276] 0426 09:56:17.681921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:18.915583 (+1233662us) service_pool.cc:225] Handling call
0426 09:56:18.938881 (+ 23298us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:18.949742 (+ 10861us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:18.949778 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:18.950212 (+ 434us) log.cc:842] Serialized 26 byte log entry
0426 09:56:19.047725 (+ 97513us) raft_consensus.cc:1631] Marking committed up to 4
0426 09:56:19.047761 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:19.047827 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:19.237654 (+189827us) raft_consensus.cc:1671] finished
0426 09:56:19.237672 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:19.237807 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":354,"raft.queue_time_us":2342,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":5345,"thread_start_us":44536,"threads_started":1,"wal-append.queue_time_us":170852}
I20240426 09:56:19.458165 9937 raft_consensus.cc:2354] T 951e4c48fcb642e58ab79652d3f86261 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 4. Current term is 5.
I20240426 09:56:19.463310 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.420764 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36762 (request call id 776) took 1843 ms. Trace:
I20240426 09:56:19.465927 10007 rpcz_store.cc:276] 0426 09:56:17.576914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:17.933863 (+356949us) service_pool.cc:225] Handling call
0426 09:56:18.129094 (+195231us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:18.511021 (+381927us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:19.420511 (+909490us) spinlock_profiling.cc:243] Waited 336 ms on lock 0x7b54000582d4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:19.420701 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":771884800}
W20240426 09:56:19.472703 10056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.129:38113, user_credentials={real_user=slave}} blocked reactor thread for 115350us
I20240426 09:56:19.470801 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af5b1ab22b40b2abe93efd45bb7676" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:19.480966 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3634dc56b34299b112ff8555d47c1e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.480966 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d3a480e831d649139ff016fb68aee3ee" is_pre_election: true
I20240426 09:56:19.490630 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f96620cc62e4bf1beefad6b59ca3968" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.513336 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa61a21874f49688b31ddd60646a658" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.511821 10054 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee; no voters:
I20240426 09:56:19.171257 12983 raft_consensus.cc:482] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:19.518510 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0131f6d1d54716ab3324104a18c9c3" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.515601 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.518193 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb385fda44e94153ae3cd9ca21240939" candidate_uuid: "f12521b6d51242629cf4297864df00b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
W20240426 09:56:19.511487 13420 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.042s sys 0.056s
I20240426 09:56:19.475364 13378 raft_consensus.cc:482] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:19.513929 13244 raft_consensus.cc:482] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:19.174341 13380 leader_election.cc:288] T ec8cf6b7af0543f3a118ab747864f366 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:19.527802 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253cc496ee9644ffba6201074808dbf0" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.528015 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e80ebfd190643a99ef02b6ea2876426" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.541908 9904 leader_election.cc:302] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a480e831d649139ff016fb68aee3ee, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:19.525310 12983 raft_consensus.cc:3037] T a7ef86a50a514e8ba09c76ca674b2100 P d3a480e831d649139ff016fb68aee3ee [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:19.395030 13248 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:19.481889 13313 raft_consensus.cc:2781] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 09:56:19.240041 13259 leader_election.cc:288] T bc3e15fcf67145f38946b62f0a5e9149 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:19.383411 13309 raft_consensus.cc:482] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d3a480e831d649139ff016fb68aee3ee)
I20240426 09:56:19.548574 13309 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:19.550474 13309 leader_election.cc:288] T 308d6ed4b68346be85caf2871916eab9 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:19.534598 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693cab52ca1f45cab4af9282a6614f90" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:19.557724 10013 raft_consensus.cc:2379] T 693cab52ca1f45cab4af9282a6614f90 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 9: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:56:19.558486 10013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e4aa133eb141699b9a5e2f0b4767d6" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:19.171378 13315 raft_consensus.cc:504] T 5c0fe10a2b3746f9909d3596580b6854 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:19.538640 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f79bd8994f4b229ca8da8638bb6d5a" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.560755 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.561842 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a202d889782b4735855ec95f82e22ac8" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:19.562484 9940 raft_consensus.cc:2354] T a202d889782b4735855ec95f82e22ac8 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for earlier term 4. Current term is 5.
I20240426 09:56:19.538661 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f4904f98c241de901e223b5ab4725d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.562873 13248 raft_consensus.cc:504] T 308d6ed4b68346be85caf2871916eab9 P d3a480e831d649139ff016fb68aee3ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:19.563519 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be50ac00c79946d89ecaf230084a401d" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.564570 9939 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a7ef86a50a514e8ba09c76ca674b2100"
dest_uuid: "f12521b6d51242629cf4297864df00b9"
mode: GRACEFUL
new_leader_uuid: "d3a480e831d649139ff016fb68aee3ee"
from {username='slave'} at 127.0.0.1:46580
I20240426 09:56:19.565085 9939 raft_consensus.cc:595] T a7ef86a50a514e8ba09c76ca674b2100 P f12521b6d51242629cf4297864df00b9 [term 5 LEADER]: Received request to transfer leadership to TS d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:19.481176 13245 raft_consensus.cc:2781] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 09:56:19.542266 9934 raft_consensus.cc:2396] T 5e80ebfd190643a99ef02b6ea2876426 P f12521b6d51242629cf4297864df00b9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 826f03b2fb9c4b9483623b041a1bf80e for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20240426 09:56:19.567988 9934 raft_consensus.cc:1261] T e117a0ad5cd34acea96de04537a80733 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Refusing update from remote peer 826f03b2fb9c4b9483623b041a1bf80e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20240426 09:56:19.533749 13420 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.776s user 0.043s sys 0.057s
I20240426 09:56:19.420136 13411 raft_consensus.cc:504] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:19.571195 13305 leader_election.cc:288] T 60bebc132e4141bdbab199326986516e P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:19.571498 13411 leader_election.cc:288] T cfa61a21874f49688b31ddd60646a658 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 9 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:19.475450 13352 raft_consensus.cc:504] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:19.572146 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0134eb2ced4f5c8e56412aaee1c225" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.533035 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8cf6b7af0543f3a118ab747864f366" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.572782 10007 raft_consensus.cc:3037] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:19.573738 13352 leader_election.cc:288] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [CANDIDATE]: Term 7 election: Requested vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:19.574378 13259 raft_consensus.cc:482] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:19.574879 13259 raft_consensus.cc:504] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:19.576941 13259 leader_election.cc:288] T f46890d4f3474e88835f7b3e08a30e1a P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:19.577387 13194 leader_election.cc:288] T ab7abcb2e80641eca676add7616cd966 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:19.577677 10086 raft_consensus.cc:2454] T a202d889782b4735855ec95f82e22ac8 P d3a480e831d649139ff016fb68aee3ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12521b6d51242629cf4297864df00b9 in term 4.
I20240426 09:56:19.546969 13313 raft_consensus.cc:482] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting leader election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:19.578529 13313 raft_consensus.cc:3037] T cf8634deb0a5425db9c47915d49386d5 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Advancing to term 8
I20240426 09:56:19.572782 9937 raft_consensus.cc:2454] T 1c0134eb2ced4f5c8e56412aaee1c225 P f12521b6d51242629cf4297864df00b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826f03b2fb9c4b9483623b041a1bf80e in term 6.
I20240426 09:56:19.579326 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7abcb2e80641eca676add7616cd966" candidate_uuid: "826f03b2fb9c4b9483623b041a1bf80e" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.580109 10007 raft_consensus.cc:2454] T a3af5b1ab22b40b2abe93efd45bb7676 P 826f03b2fb9c4b9483623b041a1bf80e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a480e831d649139ff016fb68aee3ee in term 8.
I20240426 09:56:19.268826 13306 raft_consensus.cc:504] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
W20240426 09:56:19.454082 10611 rpcz_store.cc:269] 0426 09:56:15.406494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:15.410711 (+ 4217us) service_pool.cc:225] Handling call
0426 09:56:19.143455 (+3732744us) spinlock_profiling.cc:243] Waited 2.16 s on lock 0x7b5400431554. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:19.171028 (+ 27573us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4978336640}
I20240426 09:56:19.538201 13244 raft_consensus.cc:504] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:19.584173 10085 raft_consensus.cc:2454] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 9.
I20240426 09:56:19.586208 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 1968) took 2653 ms. Trace:
I20240426 09:56:19.587795 13117 leader_election.cc:288] T ddc5aa33dc824544ade18161bf2800a6 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:19.553359 13421 raft_consensus.cc:2781] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 09:56:19.480602 13276 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:19.589865 13276 leader_election.cc:288] T 007f1451130c4dfcbd86145d56a6f67e P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), d3a480e831d649139ff016fb68aee3ee (127.8.141.131:34821)
I20240426 09:56:19.590368 13306 leader_election.cc:288] T 69700f6b120b4f4ebc0b43b14166f4c9 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:19.516156 13292 consensus_queue.cc:1046] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.027s
I20240426 09:56:19.588577 13421 raft_consensus.cc:482] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1fee3da01ebc463396397057fdb81a68)
I20240426 09:56:19.591570 13292 consensus_queue.cc:1225] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer 826f03b2fb9c4b9483623b041a1bf80e log is divergent from this leader: its last log entry 7.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20240426 09:56:19.586534 10085 rpcz_store.cc:276] 0426 09:56:16.932971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:19.585472 (+2652501us) service_pool.cc:225] Handling call
0426 09:56:19.585766 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:19.585936 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:19.585960 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:19.585998 (+ 38us) raft_consensus.cc:1631] Marking committed up to 4
0426 09:56:19.586036 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:19.586081 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:19.586163 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:19.587739 10015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1998) took 1940 ms. Trace:
I20240426 09:56:19.593487 10015 rpcz_store.cc:276] 0426 09:56:17.647823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:18.451101 (+803278us) service_pool.cc:225] Handling call
0426 09:56:18.467916 (+ 16815us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:18.528723 (+ 60807us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:18.528750 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:18.528789 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0426 09:56:18.528818 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:19.587540 (+1058722us) spinlock_profiling.cc:243] Waited 60.6 ms on lock 0x7b54002afdd4. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:19.587582 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:19.587688 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":139382144}
I20240426 09:56:19.595295 9939 raft_consensus.cc:1226] T f46890d4f3474e88835f7b3e08a30e1a P f12521b6d51242629cf4297864df00b9 [term 8 LEADER]: Rejecting Update request from peer 826f03b2fb9c4b9483623b041a1bf80e for earlier term 7. Current term is 8. Ops: []
I20240426 09:56:19.593462 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 1969) took 2660 ms. Trace:
I20240426 09:56:19.618265 10085 rpcz_store.cc:276] 0426 09:56:16.933202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:19.592775 (+2659573us) service_pool.cc:225] Handling call
0426 09:56:19.593039 (+ 264us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:19.593197 (+ 158us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:19.593220 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:19.593256 (+ 36us) raft_consensus.cc:1631] Marking committed up to 3
0426 09:56:19.593285 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:19.593330 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:19.593415 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:19.620108 10085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59762 (request call id 1971) took 2619 ms. Trace:
I20240426 09:56:19.620479 10085 rpcz_store.cc:276] 0426 09:56:17.000326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:19.619244 (+2618918us) service_pool.cc:225] Handling call
0426 09:56:19.619610 (+ 366us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:19.619791 (+ 181us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 09:56:19.619816 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:19.619860 (+ 44us) raft_consensus.cc:1631] Marking committed up to 7
0426 09:56:19.619892 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:19.619942 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:19.620052 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 09:56:19.511113 13373 raft_consensus.cc:504] T ac0131f6d1d54716ab3324104a18c9c3 P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:19.626600 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfada00f42a411f813bfc35d6c9ec9d" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:19.631601 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fd22bc1cf64e38840dcfb850cfe090" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:19.633122 10015 raft_consensus.cc:2379] T 35fd22bc1cf64e38840dcfb850cfe090 P 826f03b2fb9c4b9483623b041a1bf80e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 5: Already voted for candidate 826f03b2fb9c4b9483623b041a1bf80e in this term.
I20240426 09:56:19.589872 13336 leader_election.cc:288] T 21e4aa688b194564bef9b4398f572caf P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:19.617988 13421 raft_consensus.cc:3037] T 1c21d6dedd1441bd977f09cdcf41d436 P f12521b6d51242629cf4297864df00b9 [term 5 FOLLOWER]: Advancing to term 6
I20240426 09:56:19.624809 13306 raft_consensus.cc:482] T 472a3b5debc94412b63080f474ae4fed P d3a480e831d649139ff016fb68aee3ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f12521b6d51242629cf4297864df00b9)
I20240426 09:56:19.538317 13378 raft_consensus.cc:504] T 7a3634dc56b34299b112ff8555d47c1e P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:19.638459 13244 leader_election.cc:288] T 5945f6dc42a14b56b9e933f0c1839de1 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:19.635661 13375 leader_election.cc:288] T 5e11de7e65e544ffac5f7683e9614536 P d3a480e831d649139ff016fb68aee3ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f12521b6d51242629cf4297864df00b9 (127.8.141.129:38113), 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411)
I20240426 09:56:19.617779 13309 consensus_queue.cc:1046] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
I20240426 09:56:19.638943 10615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45564 (request call id 647) took 2792 ms. Trace:
I20240426 09:56:19.592586 12976 raft_consensus.cc:504] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false } }
I20240426 09:56:19.548010 13242 raft_consensus.cc:2781] T 6151686b1a24447880e49b7d7395fcc1 P d3a480e831d649139ff016fb68aee3ee [term 7 FOLLOWER]: Leader election won for term 7
I20240426 09:56:19.640514 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3e15fcf67145f38946b62f0a5e9149" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:19.668715 13336 raft_consensus.cc:482] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 09:56:19.672922 10054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.141.132:34313, user_credentials={real_user=slave}} blocked reactor thread for 161176us
W20240426 09:56:19.674597 10583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.141.131:34821, user_credentials={real_user=slave}} blocked reactor thread for 230081us
W20240426 09:56:19.681675 9905 leader_election.cc:334] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313): Timed out: RequestConsensusVote RPC to 127.8.141.132:34313 timed out after 1.961s (SENT)
I20240426 09:56:19.682116 9905 leader_election.cc:302] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12521b6d51242629cf4297864df00b9; no voters: 1fee3da01ebc463396397057fdb81a68, d3a480e831d649139ff016fb68aee3ee
I20240426 09:56:19.682896 12976 leader_election.cc:288] T 8574d769068f477e859574c959d5880e P f12521b6d51242629cf4297864df00b9 [CANDIDATE]: Term 10 election: Requested vote from peers 826f03b2fb9c4b9483623b041a1bf80e (127.8.141.130:44411), 1fee3da01ebc463396397057fdb81a68 (127.8.141.132:34313)
I20240426 09:56:19.683434 12976 raft_consensus.cc:2726] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 09:56:19.686101 13336 raft_consensus.cc:504] T e117a0ad5cd34acea96de04537a80733 P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12521b6d51242629cf4297864df00b9" member_type: VOTER last_known_addr { host: "127.8.141.129" port: 38113 } } peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130" port: 44411 } } peers { permanent_uuid: "d3a480e831d649139ff016fb68aee3ee" member_type: VOTER last_known_addr { host: "127.8.141.131" port: 34821 } }
I20240426 09:56:19.679862 10015 raft_consensus.cc:2379] T bc3e15fcf67145f38946b62f0a5e9149 P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a480e831d649139ff016fb68aee3ee in current term 10: Already voted for candidate f12521b6d51242629cf4297864df00b9 in this term.
I20240426 09:56:19.688203 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2caa6ec04942588af2288b0ac9ef6b" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e"
I20240426 09:56:19.688308 12976 consensus_queue.cc:1046] T 472a3b5debc94412b63080f474ae4fed P f12521b6d51242629cf4297864df00b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1fee3da01ebc463396397057fdb81a68" member_type: VOTER last_known_addr { host: "127.8.141.132" port: 34313 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.000s
W20240426 09:56:19.689533 10586 connection.cc:232] RPC call timeout handler was delayed by -1.0521182799999451s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1734s for a response
I20240426 09:56:19.691685 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34390 (request call id 1999) took 2039 ms. Trace:
W20240426 09:56:19.692359 10083 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48202 (request call id 626) took 3302 ms (client timeout 1774 ms). Trace:
I20240426 09:56:19.693192 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47248 (request call id 2473) took 2814 ms. Trace:
W20240426 09:56:19.705744 10083 rpcz_store.cc:269] 0426 09:56:16.390115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:16.393122 (+ 3007us) service_pool.cc:225] Handling call
0426 09:56:19.692194 (+3299072us) spinlock_profiling.cc:243] Waited 1.97 s on lock 0x7b5400251054. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c4a43e 00007f9ca7b88897 00007f9ca126cbd4 00007f9ca126cb62 00007f9ca126ca91 00007f9ca126ca0c 00007f9ca126bcc2 00007f9ca106da4c 00007f9ca106ce86 00007f9ca106c82f
0426 09:56:19.692296 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4530771328}
I20240426 09:56:19.707540 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d051be8c743a18cd550e145b6564e" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9"
I20240426 09:56:19.673873 13360 raft_consensus.cc:3032] T 67dfdb63e28a460486dfe163f03bf820 P 1fee3da01ebc463396397057fdb81a68 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 09:56:19.705924 10586 leader_election.cc:302] T 951e4c48fcb642e58ab79652d3f86261 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters: 826f03b2fb9c4b9483623b041a1bf80e
I20240426 09:56:19.695412 10015 raft_consensus.cc:3037] T 9c2caa6ec04942588af2288b0ac9ef6b P 826f03b2fb9c4b9483623b041a1bf80e [term 10 FOLLOWER]: Advancing to term 11
I20240426 09:56:19.705173 10011 rpcz_store.cc:276] 0426 09:56:17.652456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:18.502357 (+849901us) service_pool.cc:225] Handling call
0426 09:56:18.502670 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 09:56:18.635229 (+132559us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 09:56:18.759109 (+123880us) log.cc:842] Serialized 14 byte log entry
0426 09:56:18.905655 (+146546us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 09:56:18.905705 (+ 50us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:56:18.905744 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:19.690959 (+785215us) spinlock_profiling.cc:243] Waited 3.48 ms on lock 0x7b180042d300. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca8bbd4b1 00007f9ca8bca231 00007f9c9fce97c6 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59
0426 09:56:19.691482 (+ 523us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b540025f654. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:19.691526 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:19.691633 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2542,"prepare.queue_time_us":121992,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":28,"raft.queue_time_us":78,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":6135,"spinlock_wait_cycles":312339072,"thread_start_us":263438,"threads_started":2,"wal-append.queue_time_us":192792}
I20240426 09:56:19.706116 9938 rpcz_store.cc:276] 0426 09:56:16.878523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 09:56:16.935448 (+ 56925us) service_pool.cc:225] Handling call
0426 09:56:16.935725 (+ 277us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 09:56:17.318286 (+382561us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 09:56:17.318314 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 09:56:17.326827 (+ 8513us) log.cc:842] Serialized 26 byte log entry
0426 09:56:17.586483 (+259656us) raft_consensus.cc:1631] Marking committed up to 6
0426 09:56:17.586519 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 09:56:19.692825 (+2106306us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b54001aed54. stack: 00007f9c9fce9863 00007f9c9fce9760 00007f9c9f64e4e4 00007f9ca9363924 00007f9ca93638b9 00007f9ca2bcb5f1 00007f9ca2c470e1 00007f9ca2c45b2e 00007f9ca7b87f59 00007f9ca126a654 00007f9ca126a5e2 00007f9ca126a511 00007f9ca126a48c 00007f9ca1269742 00007f9ca106da4c 00007f9ca106ce86
0426 09:56:19.692871 (+ 46us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 09:56:19.692917 (+ 46us) raft_consensus.cc:1671] finished
0426 09:56:19.692928 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 09:56:19.693121 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":60698,"raft.queue_time_us":112583,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":8617,"spinlock_wait_cycles":602635008,"thread_start_us":247237,"threads_started":1,"wal-append.queue_time_us":380697,"wal-append.run_cpu_time_us":1525,"wal-append.run_wall_time_us":1006067}
I20240426 09:56:19.697767 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed47e1980f4a14a46cc61ea156eb53" candidate_uuid: "1fee3da01ebc463396397057fdb81a68" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f12521b6d51242629cf4297864df00b9" is_pre_election: true
I20240426 09:56:19.711587 9933 raft_consensus.cc:2454] T c6ed47e1980f4a14a46cc61ea156eb53 P f12521b6d51242629cf4297864df00b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fee3da01ebc463396397057fdb81a68 in term 8.
I20240426 09:56:19.711189 10586 leader_election.cc:302] T 6151686b1a24447880e49b7d7395fcc1 P 1fee3da01ebc463396397057fdb81a68 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fee3da01ebc463396397057fdb81a68, f12521b6d51242629cf4297864df00b9; no voters:
I20240426 09:56:19.711166 10011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da46d724daf4bbf9c49a393e3579156" candidate_uuid: "d3a480e831d649139ff016fb68aee3ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "826f03b2fb9c4b9483623b041a1bf80e" is_pre_election: true
I20240426 09:56:19.713111 13128 raft_consensus.cc:482] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:56:19.713652 13128 raft_consensus.cc:504] T 007f1451130c4dfcbd86145d56a6f67e P d3a480e831d649139ff016fb68aee3ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "826f03b2fb9c4b9483623b041a1bf80e" member_type: VOTER last_known_addr { host: "127.8.141.130"